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0\ . Abstract 

(N 

Under-determined systems of linear equations with sparse solutions have been the subject of an extensive 

research in last several years above all due to results of [9, 10, 21]. In this paper we will consider noisy 

under-determined linear systems. In a breakthrough [10] it was established that in noisy systems for any 
Y^ I linear level of under-determinedness there is a linear sparsity that can be approximately recovered through 

an SOCP (second order cone programming) optimization algorithm so that the approximate solution vector 

is (in an ^2-norm sense) guaranteed to be no further from the sparse unknown vector than a constant times 
^ i the noise. In our recent work [53] we established an alternative framework that can be used for statistical 

O I performance analysis of the SOCP algorithms. To demonstrate how the framework works we then showed 

in [53] how one can use it to precisely characterize the generic (worst-case) performance of the SOCP. In 
O ■ this paper we present a different set of results that can be obtained through the framework of [53]. The 

T^lj- \ results will relate to problem dependent performance analysis of SOCP's. We will consider specific types 

O ■ of unknown sparse vectors and characterize the SOCP performance when used for recovery of such vectors. 

We will also show that our theoretical predictions are in a solid agreement with the results one can get 

through numerical simulations. 



'^ \ Index Terms: Noisy systems of linear equations; SOCP; ^i -optimization; compressed sensing. 



1 Introduction 

In recent years there has been an enormous interest in studying under-determined systems of linear equations 
with sparse solutions. With potential applications ranging from high-dimensional geometry, image recon- 
struction, single-pixel camera design, decoding of linear codes, channel estimation in wireless communi- 
cations, to machine learning, data-streaming algorithms, DNA micro-arrays, magneto-encephalography etc. 
(see, e.g. [3,8, 11, 18,26,38,42,46-49,56,65,67] and references therein) studying these systems seems 
to be of substantial theoretical/practical importance in a variety of different area. In this paper we study 
mathematical aspects of under-determined systems and put an emphasis on theoretical analysis of particular 
algorithms used for solving them. 

In its simplest form solving an under-determined system of linear equations amounts to finding a, say, 
/c-sparse x such that 

Ax = y (1) 

where A is an m x n (tti < n) matrix and y is an m x 1 vector (see Figure 1 ; here and in the rest of the 



paper, under A;-sparse vector we assume a vector that has at most k nonzero components). Of course, the 
assumption will be that such an x exists. To make writing in the rest of the paper easier, we will assume the 
so-called linear regime, i.e. we will assume that k = /Sn and that the number of equations is tti = an where 
a and /3 are constants independent of n (more on the non-linear regime, i.e. on the regime when m is larger 
than Unearly proportional to k can be found in e.g. [17, 30, 31]). 
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Figure 1 : Model of a linear system; vector x is A;-sparse 

Clearly, if A; < ^ the solution is unique and can be found through an exhaustive search. However, in the 
linear regime that we have just assumed above (and will consider throughout the paper) the exhaustive search 
is clearly of exponential complexity. Instead one can of course design algorithms of much lower complexity 
while sacrificing on the recovery abilities, i.e. on the recoverable size of the nonzero portion of vector x. 
Various algorithms have been introduced and analyzed in recent years throughout the literature from those 
that relate to the parallel design of matrix A and the recovery algorithms (see, e.g. [1, 34, 35, 39,45, 68]) 
to those that assume only the design of the recovery algorithms (see, e.g. [10, 19, 21, 25, 43, 44, 62, 63]). 
If one restricts to the algorithms of polynomial complexity and allows for the design of A then the results 
from [1, 39, 45] that guarantee recovery of any /c-sparse x in (1) for any < a < 1 and any /3 < ^ are 
essentially optimal. On the other hand, if one restricts to the algorithms of polynomial complexity and does 
not allow for the parallel design of A then the results of [10,21] established that for any a > there still 
exists a /3 > such that any k = /3n-sparse x in (1) can be recovered via a polynomial time Basis pursuit 
(BP) algorithm. 

Since the BP algorithm is fairly closely related to what we will be presenting later in the paper we 
will now briefly introduce it (we will often refer to it as the ^i -optimization concept; a slight modifica- 
tion/adaptation of it will actually be the main topic of this paper). Variations of the standard £i -optimization 
from e.g. [12, 16,51] as well as those from [28,32,50] related to ^^-optimization, < q < 1, aie possible 
as well; moreover they can all be incorporated in what we will present below. The ^i -optimization con- 
cept suggests that one can maybe find the /c-sparse x in (1) by solving the following ^i-norm minimization 
problem 



mm ||x||i 
subject to A:x. = y. 



(2) 



As mentioned above the results from [10, 21] were instrumental in theoretical characterization of (2) and 
its popularization in sparse recovery and even more so in generating an unprecedented interest in sparse 
recovery algorithms. The main reason is of course the quality of the results achieved in [10,21]. Namely, [10] 
established that for any a > there is a /3 > such that the solution of (2) is the k = /3n-sparse x in (1). 
In a statistical and large dimensional context in [21] and later in [55,57] for any given value of a the exact 
value of the maximum possible /3 was determined. 



The above sparse recovery scenario is in a sense idealistic. Namely, it assumes that y in (2) was obtained 
through (1). On other hand in many applications only a noisy version of Ax. may be available for y (this 
is especially so in measuring type of applications) see, e.g. [10,33,66] (another somewhat related version 
of "imperfect" linear systems are the under-determined systems with the so-called approximately sparse 
solutions; more in this direction can be found in e.g. [10,59]). When that happens one has the following 
equivalent to (1) (see. Figure 2) 

y = ^x + V, (3) 

where v is an m x 1 so-called noise vector (the so-called ideal case presented above is of course a special case 
of the noisy one given in (3)). Finding the A;-sparse x in (3) is now incredibly hard. In fact it is pretty much 
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Figure 2: Model of a linear system; vector x is fc-sparse 

impossible. Basically, one is looking for a /c-sparse x such that (3) holds and on top of that v is unknown. 
Although the problem is hard there are various heuristics throughout the literature that one can use to solve 
it approximately. Majority of these heuristics are based on appropriate generalizations of the corresponding 
algorithms one would use in the noiseless case. Thinking along the same lines as in the noiseless case one 
can distinguish two scenarios depending on the availability of the freedom to choose/design A. If one has 
the freedom to design A then one can adapt the corresponding noiseless algorithms to the noisy scenario 
as well (more on this can be found in e.g. [5]). However, in this paper we mostly focus on the scenario 
where one has no control over A. In such a scenario one can again make a parallel to the noiseless case 
and look at e.g. CoSAMP algorithm from [43] or Subspace pursuit from [20]. Essentially, in a statistical 
context, these algorithms can provably recover a linear sparsity while maintaining the approximation error 
proportional to the norm-2 of the noise vector which is pretty much a benchmark of what is currently known. 
These algorithms are in a way perfected noisy generalizations of the so-called Orthogonal matching pursuit 
(OMP) algorithms. 

On the other hand in this paper we will focus on generalizations of BP that can handle the noisy case. 
To introduce a bit or tractability in finding the fc-sparse x in (3) one usually assumes certain amount of 
knowledge about either x or v. As far as the tractability assumptions on v are concerned one typically (and 
possibly fairly reasonably in applications of interest) assumes that ||v||2 is bounded (or highly likely to be 
bounded) from above by a certain known quantity. The following second-order cone programming (SOCP) 
analogue to (or say noisy generalization of) (2) is one of the approaches that utilizes such an assumption 
(more on this approach and its variations can be found in e.g. [10]) 



mm ||x||i 

X 

subject to ||y — Ax||2 < r. 



ocp 



(4) 



where rgocp is a quantity such that ||v||2 < Vgocp (or rsocp is a quantity such that ||v||2 < rgocp is say highly 



likely). For example, in [10] a statistical context is assumed and based on the statistics of v, Vsocp was chosen 
such that ||v||2 < Vsocp happens with overwhelming probability (as usual, under overwhelming probability 
we in this paper assume a probability that is no more than a number exponentially decaying in n away from 
1). Given that (4) is now among few almost standard choices when it comes to finding an approximation to 
the /c-sparse x in (3), the literature on its properties is vast (see, e.g. [10,24,33,61] and references therein). 
We here briefly mention only what we consider to be the most influential work on this topic in recent years. 
Namely, in [10] the authors analyzed performance of (4) and showed a result similar in flavor to the one that 
holds in the ideal - noiseless - case. In a nutshell the following was shown in [10]: let x be a /3n-sparse 
vector such that (3) holds and let yisocp be the solution of (4). Then 

where /3 is a constant independent of n and C is a constant independent of n and of course dependent on a 
and /3. This result in a sense establishes a noisy equivalent to the fact that a linear sparsity can be recovered 
from an under-determined system of linear equations. In an informal language, it states that a linear sparsity 
can be approximately recovered in polynomial time from a noisy under-determined system with the norm of 
the recovery error guaranteed to be within a constant multiple of the noise norm (as mentioned above, the 
same was also established later in [43] for CoSAMP and in [20] for Subspace pursuit). Establishing such a 
result is, of course, a feat in its own class, not only because of its technical contribution but even more so 
because of the amount of interest that it generated in the field. 

In our recent work [53] we designed a framework for performance analysis of the SOCP algorithm 
from (4). We then went further in [53] and showed how the framework practically works through a precise 
characterization of the generic (worst-case) performance of the SOCP from (4). In this paper we will again 
focus on the general framework developed in [53]. This time though we will focus on the. problem dependent 
performance analysis of the SOCP. In other words, we will consider specific types of unknown sparse vectors 
X in (3) and provide a performance analysis of the SOCP when applied for recovery of such vectors. 

Before going into the details of the SOCP approach we should also mention that the SOCP algorithms 
are of course not the only possible generalizations (adaptations) of ii optimization to the noisy case. For 
example, LASSO algorithms (more on these algorithms can be found in e.g. [7, 14, 15,41, 60, 64] as well as 
in recent developments [4,22,52]) are a very successful alternative. In our recent work [52] we established 
a nice connection between some of the algorithms from the LASSO group and certain SOCP algorithms 
and showed that with respect to certain performance measure they could be equivalent. Besides the LASSO 
algorithms the so-called Dantzig selector introduced in [13] is another alternative to the SOCP algorithms 
that is often encountered in the literature (more on the Dantzig selector as well as on its relation to the 
LASSO or SOCP algorithms can be found in e.g. [2,6,27,29,36,37,40]). Depending on the scenarios they 
are to be applied in each of these algorithms can have certain advantages/disadvantages over the other ones. 
A simple (general) characterization of these advantages/disadvanatges does not seem easy to us. In a rather 
informal language one could say that LASSO and SOCP are expected to perform better (i.e. to provide a 
solution vector that is under various metrics closer to the unknown one) in a larger set of different scenarios 
but as quadratic programs could be slower than the Dantzig selector which happens to be a linear program. 
Of course, whether LASSO or SOCP algorithms are indeed going to be slower or not or how much larger 
would be a set of different scenarios where they are expected to perform better are interesting/important 
questions. While clearly of interest answering these questions certainly goes way beyond the scope of the 
present paper and we will not pursue it here any further. 

Before we proceed with the exposition we briefly summarize the organization of the rest of the paper. 
In Section 2, we recall on a set of powerful results presented in [53] and discuss further how they can be 
utilized to analyze the problem dependent performance of the SOCP from (4). The results that we will 
present in Section 2 will relate to the so-called general sparse signals x. In Section 3 we will show how 



these results from Section 2 that relate to the general sparse vectors x can be specialized further so they 
cover the so-called signed vectors x. Finally, in Section 4 we will discuss obtained results. 

2 SOCP's problem dependent performance - general x 

In this section we first recall on the basic properties of the statistical SOCP's performance analysis frame- 
work developed in [53]. We will then show how the framework can be used to characterize the SOCP from 
(4) in certain situations when the SOCP's performance substantially depends on x. 

2.1 Basic properties of the SOCP's framework 

Before proceeding further we will now first state major assumptions that will be in place throughout the 
paper (clearly, since we will be utilizing the framework of [53] a majority of these assumptions was already 
present in [53]). Namely, as mentioned above, we will consider noisy under-determined systems of linear 
equations. The systems will be defined by a random matrix A where the elements of A are i.i.d. standard 
normal random variables. Also, we will assume that the elements of v are i.i.d. Gaussian random variables 
with zero mean and variance a. x will be assumed to be the original x in (3) that we are trying to recover (or 
a bit more precisely approximately recover). We will assume that x is any A;-sparse vector with a given fixed 
location of its nonzero elements and a given fixed combination of their signs. Due to assumed statistics the 
analysis (and the performance of (4)) will clearly be irrelevant with respect to what particular location and 
what particular combination of signs of nonzero elements are chosen. We will therefore for the simplicity of 
the exposition and without loss of generality assume that the components xi, X2, . . . , ^n-k of x are equal 
to zero and the components ^n-k+i,^n-k+2, . . . ,x„ of x are greater than or equal to zero. Moreover, 
throughout the paper we will call such an x /c-sparse and positive. In a more formal way we will set 

Xi = X2 = • • • = X„_fc = 

X„-fc+l > 0, ±n-k+l > 0, . . . , x„ > 0. (6) 

We also now take the opportunity to point out a rather obvious detail. Namely, the fact that x is positive is 
assumed for the purpose of the analysis. However, this fact is not known a priori and is not available to the 
solving algorithm (this will of course change in Section 3). 

Before proceeding further we will introduce a few definitions that will be useful in formalizing/presenting 
our results as well as in conducting the entire analysis. Following what was done in [53] let us define the 
optimal value of a slightly changed objective of (4) in the following way 

fobj{cr,x,A,v,rsocp) = mill ||x||i - ||x||i 



X 



subject to ||y - Ax||2 < Vgocp- (7) 

To make writing easier we will instead of fobj{<^, x, A, v, rsocp) write just fobj- A similar convention will be 
applied to few other functions throughout the paper. On many occasions, though, (especially where we deem 
it as substantial to the presentation) we will also keep all (or a majority of) arguments of the corresponding 
functions. 

Also let Xsocp be the solution of (4) (or the solution of (7)) and let "Wsocp G ^" be the so-called error 
vector defined in the following way 

^^socp — ^socp X. (o) 

Our main goal in this paper will then boil down to various characterizations of w^ocp and fobj- Throughout 
the paper we will heavily rely on the following theorem from [53] that provides a general characterization 



oi Wsocp and fobj. 

Theorem 1. ( [53] — SOCP's performance characterization) Let v be an n x 1 vector ofi.i.d. zero-mean 
variance a^ Gaussian random variables and let Abe an m x n matrix ofi.i.d. standard normal random 
variables. Further, let g and hbe m x 1 and n x 1 vectors ofi.i.d. standard normals, respectively and let z 
be n X 1 vector of all ones. Consider a k-sparse x defined in (6) and a y defined in (3) for x = x. Let the 
solution of (4) be Xgocp and let the so-called error vector of the SOCP from (4) be Wgocp = ^socp — x. Let 
fsocp in (4) be a positive scalar Let n be large and let constants a = — and fiw = - be below the following 
so-called fundamental characterization of Hi optimization 

R -(erfinv(\^^))'^ 
(1 - P^)^ ^erfinv{—^) = 0. (9) 

Consider the following optimization problem: 



i/,X 



Cprim(o-, g,h,x, r^ocp) = max (TA/||g|||i/2 - ||z/h + z - A||^ - ^ AjX 



lyr. 



socp 
i=n—k+l 



subject to z/ > 

< Aj < 2, 1 < z < n. (10) 

Let i) and A be the solution of (10). Set 

II - II ||i>h + z-A||2 

||w||2=(T =■ (11) 

^||g||2l>2-||z>h + Z-A||2 

Then: 

P(||x + Wsocp||i - ||x||i G (£',fprim(cr,g, h,x, r^ocp)) -ef°''^^|£',^prim(cr,g, h,x, r^ocp))!, 

E^prim{(T, g, h, X, rsocp)) + ei^'\E^prim{a, g, h, X, rsocp))\) = 1 - e~^2 " (12) 

and 



P((l - 6S^°^^))i?||w||2 < Wv^socph < (1 + e(^°^^))i^||w||2) = 1 - e-4^°"'", (13) 

where e^"'^^' > is an arbitrarily small constant and ^2^^' is a constant dependent on e^"'^^' and a but 
independent of n. 

Proof. Presented in [53]. D 

Remark: A pair (a, 13^) lies below the fundamental characterization (9) if a > a^ and a^ and j3w are such 
that (9) holds. 

2.2 Problem dependent properties of the framework 

To facilitate the exposition that will follow we similarly to what was done in [53,54,57] set 

where [|h|U, |h|U, . . . , |h|^^_ J] are the magnitudes of [hi, h2, . . . , h„_fc] sorted in increasing order and 
\}^nlk+iMaIk+2^ • • • ' '^^ ] ^^ th^ elements of [-h„_fe+i, -h„_fc+2, • • • , -h„] sorted in decreasing order 



(possible ties in the sorting processes are of course broken arbitrarily). One can then rewrite the optimization 
problem from (10) in the following way 



Cp«m(cr, g, h,x, r^ocp) = max ^ \ \\g,\\lv'^ - Wuh - 2 + \\\l - "S^ \i±i-ursocp 

u,X » ^ — ' 

i=n—k+l 

subject to u > 

0< Xi<l,l<i<n-k 

0< Xi<2,n-k + l<i<n. (15) 

In what follows we will restrict our attention to a specific class of unknown vectors x. Namely, we will 
consider vectors x that have amplitude of the nonzero components equal to Xmag- In the noiseless case these 
problem instances are typically the hardest to solve (at least as long as one uses the ii optimization from 
(2)). We will again emphasize that the fact that magnitudes of the nonzero elements of x are Xmag is not 
known a priori and can not be used in the solving algorithm (i.e. one can not add constraints that would 
exploit this knowledge in optimization problem (4)). It is just that we will consider how the SOCP from 
(4) behaves when used to solve problem instances generated by such an x. Also, for such an x (15) can be 
rewritten in the following way 



CrZli^^S,'^,Xmag,rsocp) =T^a.X aJ \\g\\'^l/'^ - \\uh - Z + XW^ - X^ag V] Xi - UTsocp 

i=n—k+l 

subject to u > 

0< Xi< 1,1 <i<n-k 

0< Xi<2,n-k + l<i<n. (16) 

Now, let i/dep and A*^^"^^^ be the solution of (16). Then analogously to (11) we can set 

llWdeplb = 0- ^ ^ _ (17) 

MWdep-\Wdeph-Z + X('i-P)\\l 

In what follows we will determine ||wrfep||2 and CLimv*^' S' '^' Xmag,rsocp) or more precisely their concen- 
trating points EWwdeph and E^^pJl^^{a,g,h,Xmag,rsocp)- All other parameters such as i^dep, A^'^^p) can 
(and some of them will) be computed through the framework as well. We do however mention right here 
that what we present below assumes a fair share of familiarity with the techniques introduced in our earlier 
papers [52,53,57]. To shorten the exposition we will skip many details presented in those papers and present 
only the key differences. 

We proceed by following the line of thought presented in [53, 57]. Since A^''^^) is the solution of (16) 
there will be parameters ci, C2, and C3 such that 

^(dep) ^ [^(^ep) ^(.ep) ^gep) , Q, 0, . . . , 0, A^f^^J , A^f/^ , . . . , A^t 2, 2, . . . , 2] 

and obviously ci <n — k, n — k<C2 < n, and < C3 < n — C2. At this point let us assume that 
these parameters are known and fixed. Then following [53,57] the optimization problem from (16) can be 



rewritten in the following way 

I = " 

?prjm(<^) g) h, X, r^ocpj = max CrW ||g||2^' — Wvilcx+l-.n ~ '^c\ + l:n ~\~ ^c\+l:n\\2 ~ ^mag / ^ \ ~ l^1"socp 

l=?l— fe+1 

subject to u > 

< Aj < 2, C2 < i < n — C3 

Xi = 2,n — C3 + l<i<n 

Xi = 0,ci + l<i<n-k. (18) 

To make writing of what will follow somewhat easier we set 



^(°^J) = O-y ||g|||z^2 _ ||jyhci+l:„ - Zci+l:n + Aci+l:n||i " X^ag ^ Aj - Z/r^ocp- (19) 

j=n— fc+1 

We then proceed by solving the optimization in (18) over u and Xci+i-.n- To do so we first look at the 
derivatives with respect to Aj, C2 + 1 < i < n — C3, of the objective in (18). Computing the derivatives and 
equalling them to zero gives 

— 77 — = 0, C2 + 1 < i < n - C3 
dXi 

— (i/hj — Zj + Xi) 

q- ; _ - X^ag = 0, C2 + 1 < i < n - C3 

|||Oo||i \ ||0 

ISlb^ ~ ||Z^llc^+l:„ — Zci+l:n + Aci + l;n||2 



Xi -Zj + Uhi = ^\/\\g\\2^^ - \\iyhc-^ + l:n " Zci + 1;„ + Aci + l:„|||, C2 + I < i < U - C3 



<j=^ X- = V]flJ\\g\\2jy2 _ ||,yhci + l;„ - Zci+l;„ + Aci+l:n|l2 + ^i -uhi,C2 + 1 < « < n - C3. 

(20) 
From the second to last line in the above equation one then has 

„2 



"'2"UISIl2^" ~ W^l^^ci + l-.cz - Zci + l:C2 ^ ^ci + l:C2ll2 



{Xi-Zi + iyhif = -^{\\g\\liy^ -\\iyhc-^+i.,c2-Zcj_+i:c2 + \ - "^ 



~ \\^"-C2+\:n—Oi ~ 2;c2+l:n— C3 II2 ~ H^tln— C3+l:n + Z„_c3+l:n||2 j (21) 

and after an easy algebraic transformation 



xi 



(Aj — Zj + Z/hj) — „ —^ (IISL^ — ||l^hc^ + l:C2 — Z;ci + l:c2l|2 ~ Il^h„_c3 + I:n + Zn_c3 + 1 " 

CJ^ + (n - C2 - C3)x4„3 



n\\2)- 

(22) 



Using (22) we further have 



|g|l2^^ ~ ||i^hc^+l:?i — Zci + l;n + Aci + l;n||2 



ISll2^ ||'^'lci+l:C2 ^ci+l:C2\\2 \\^"-C2+l.n—C3 ^C2+l.n—c-s ~r ■^C2+l:n—cs\\2 11^^!^— C3+I:n + ^n— C3+I:n|l2 



\\ ) n \\ 1 1 1 y 1 1 1 11/ 

..Slb'^ ||^^ci + l:c2 Zci + l:c2ll2 ~ || ^^n— C3 + l:n + '^n—C3-^l:n\\2- 

Cr2 + (n - C3 - C2)x'^^ag " 



(23) 



Plugging the value for Aj from (18) in (19) gives 

/ = " 

^(obj) ^ a^J\\g\\lu'^ -\\uhc^+i.n-Zci+l:n + Xci+l;n\\l-Xrnag ^ K - VVsocp 

i=n—k+l 

2 I /^ ^ „ \r„2 



(T^ + (n - C3 - C2)x. 



!!l^./llo.||2„2_ ||„U .. -r, ,, I \ ,, 112 



a 

n~C3 



S\\2'^ ~ \W^ci + l:n — Zci+l:n + Acj+l:n|l2 



XmagC^' C3 C2J + VXfnag / ^ "^i ^'^S^^mag ^fsocp- \^^} 

i=C2+l 



Combining (23) and (24) we finally obtain 



&''^'' = \/(^^ + {'^-C3-C2)xl^ag\/\\s\\W- \Whc,+l:c2 - Z^ + hca ||i - \\uhn-c3 + l:n + Zn^C3+l:n\\l 

n— C3 

^ysocp XfYiag / ^ "-i) Xmagyfl C3 C2J J^C^Xffiag- v-^J) 

J=C2 + 1 

Equalling the derivative of ^^°^^> with respect to v to zero further gives 
dv 



(^a2 + (n - C3 - C2)x2^„g) ^^||g||ii^2- ||j^hci + l:c2 -Zci + l:c2ll2- ||z^h„_c3 + l:„ + Z„_c3 + l:„||i 

n-C3 

Csocp Xmag / ^ "-ij — L). 
i=C2+l 

(26) 



Let 




^dep ~ 


- hcj + i.c2Zci + l;C2 - h„_, 




C2 n 


O-dep ~ 


- E hf+ E h; 




j=ci + l i=n— C3 + I 




n— C3 


fdep - 


- 1"socp Xmag / ^ ilj 




i=C2+l 



C3+l:n'^n.-C3+l:n 



O'dep 



g|li - (E?ie,+1 h? + Er=n-e3+l ^D V^' + (^ " ^3 " C2)x2,,^(||g||i - d,ep 



0-2 + (n - C3 - C2)x2^a3 r^ep 



^dep 



9dep = 1 = • K^l) 

0-2 + (n - C3 - C2)xl,ag rdep ^'^"^ 

Then combining (26) and (27) one obtains 

{adepV + hdep) = llsL'^ — ||'^hc^+l:c2 — Zci + l;c2ll2 ~ ll^^^ra-C3+l:n + Z„_c3+l:n||2- (28) 

After solving (28) over u we have 

~\0'depOdep ~ ^dep) ~ \ {fldep^dep ~ ^dep) ~ \Pdep ' ll^ci + l;C2 lb "•" ll^»i— C3 + l;n|l2J v'^t^ep ~ llSib ' ^dep) 

^ = 2 [r~FT2 — ; • 

'^dep- mh +ddep 

(29) 
Following what was done in [53, 57], we have that a combination of (20) and (29) gives the following three 
equations that can be used to determine ci, C2, and ca (the equations are rather inequalities; since we will 
assume a large dimensional scenario we will instead of any of the inequalities below write an equality; this 
will make writing much easier). 



r , -^niag /111129 11 T ,\ Ii2 n 

Uh.c2 - Zc2 H V I|g|l2^ - \\vaci+l:n " Zci + l;n + Aci+l:„||2 = 



r I I ^mag / \\ \\2 2 W v. i\ Il2 n 

Z^nn-C3 + Z„_C3 H V llSlb^ — ||J^nci+l:n — Zci + l;n + Aci+l:n||2 — U 

CT * 

~V^depbdep ~ ^dep) ~ \ y^dep^dep ~ ^dep) ~ \Pdep ' ll^ci+l:C2 II2 ' \\'^n—c-i+l:n\\2)v^dep ~ llSib ' ^dep) 

^=1 ;^2 ii„ii2 I J ^ -*-• 

(30) 
The last term that appears on the right hand side of the first two of the above equations can be further 
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simplified based on (23) in tlie following way 



|g|l2^^ ~ ||i^hc^+l:ri — Zcj + l;n + Aci + l;n|l2 



'^\ llSlb'^ ~ lFnci + l;C2 — '^ci + l:C2\\2 ~ \W"-n~c-i+l:n + Zn-C3+I:n|l2 



(T^ + (n — C3 — C2)x2 



mag 



C^Vllglli^^ - ^"^ddep + ^USdep - (llZci+lxalli + ||z„_c3+l:„||l) Cr./||g|||zy2 - v'^ddep + ^VSdep - (C2 - Ci + C3 



Cr2 + (n - C3 - C2)x^„g Jcj2 + (n - C3 - C2)X^„^ 



where we of course recognized that ||zci+i:c2ll2 + llzn-cs+imlli = C2 — ci + C3. Combining (27) and (31) 
one can then simplify the equations from (30) in the following way 



cr2 + (n - C3 - C2)xl,^g 



Z/hca-ZcjH , """^ =A/||g||2'^^ -Z^^<^dep + 2t/gdep- (C2 - Ci +C3) = 



Vh.n-c'i + 2n~C3 + —j= """^ yllgllii^^ -i^^(idep + 2;^gdep- (C2 -Ci +C3) = 

Y'cj2 + (n - C3 - C2)x2^ag 

-{adepbdep - Sdep) " J {adepbdep " ^dep)^ " (&lep + (^2 " Cl + C3))(a^gp - ||g||| + ddep) 

ci 2 II Til i J ~ 

aip-||g|l2+'^rfep 

(32) 
Let ci, C2, and 63 be the solution of (32). Then 

_ -{odTpbdep - SdTp) - \l {Od^phdep - Sd^p)"^ - {bdep + (C2 - cl + Cs)){ad!^'^ - \\g\\l + ddep) 

'^d-^P ~ 2 _ II ||2 , J ■ ' 

(^dep I|g|l2 "•" "'dep 

^ ^ (33) 

where s^, ddep, a^p, and bdep are Sdep, ddep, adep, and bdep from (27) computed with ci, c^, and q. From 
(17) one then has 

1 1 r •, (dep) 1 1 

llwrfeplb = cr , ^ =■ (34) 

/llrrl|2,,2 11, , U^ „_ _i_ \(aep) ||2 

V llglb'^dep - \\J^depnei+l:n " Zcl+1:„ + A^^^^^^.^Hs 

Combining (31) and (34) one further has 



r2 



„ „ , M2''depin-C3-C2)^^ + ulddep-2us;;irp + {c2-Ci+C3) 

llwrfeplb = cr- , • (35) 

V llslli^ip - '^depddep + ^l^depSlTp - {c2 - Ci + C3) 

Combination of (33) and (35) is conceptually enough to determine ||W(iep||2 (and then afterwards easily 

^^prlm i^^ g' ^' ^mag , fsocp))- What is left to be done is a computation of all unknown quantities that appear 
in (33) and (35). We will below show how that can be done. As mentioned earlier what we will present 
substantially relies on what was shown in [53,57] and we assume a familiarity with the procedures presented 
there. 

The first thing to resolve is (32). Since all random quantities concentrate we will be dealing (as in 
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(31) 



[53,57]) with the expected values. To compute the solution of (32), ci, 62, and 63, we will need the following 
expected values 

ni<vii2 niVi I|2 niu 112 tpuv, ii2 

-'^Il&ll2'-'^ll'^ci+l:ra-fc|l2i ^ ll'^ra-fc+l:C2 112' -'^ll^n~C3+l:n||2) 

Ei\^ + l:n~.k'^Ci + l:n-k),E{h^^^.^-i^.^^Zn^k + l:C2)i -E'(h„_c^^_i.„Z„_c3 + l:„). (36) 

Clearly, since components of g are i.i.d. standard normals one easily has 

E\\g\\l = m. (37) 

Let ci = (1 — 6i)n, C2 = 62n, and C3 = 6311 where Oi, 62, and ^3 are constants independent of n. Then as 
shown in [53,57] 



l:z^\\2 






where we of course recall that (3^ = -■ Also, as shown in [53, 57] 

EWh l.^^ \\l B /\/2erfinv(2i^-l)\ 
«^^ n V2^ \^ ^(erfinv(2i^-i))2 j 

and 

-Ellh _ui l|2 /3 /\/2erfinv(2^^^-l)\ 

lim ^ll^»-^3+l:n|l2 ^ J^ ^ ^ \ ^; ^ +^3, (40) 

n->^ n ^2^ \^ ^(erfinv(2^^-i))2 y 

Following further what was established in [53,57] we have 

lim ^(^^i+i-»-^^-^+^--^) = V _ ^^) /2;^-(eriinv(i^))A 
n-5>oo n \ V vr / 

j.^ ^(hLfc+l:c.^n-fe+l:c.) ^ / ^ /T^„(erfinV(2i^-l))A 

n^oo n \ V 27r / 

lim ^(^»-e3+l:nZn-e3+l:n) ^ _ V /T^-(erfinV(2i^-l))A ^^^^ 

n-5>oo n \ V 27r / 

From (41) we also have 

lim ^^^-'^^^^-^ = {p^.[^e-^^^^'^''^-'A - f/3../Te-(-finv(2i^^i))A ^4^) 
n^oo n \V27r I \ \ 2iT I 

The only other thing that we will need in order to be able to compute ci, 62, and 63 (besides the expectations 
from (36)) are the following inequalities related to the behavior of hd, hcj, and hc^. Again, as shown 
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P(^/2erfinv((l + e^)(^-^))<h,J < e'^^" 

-L Pin 



in [53,57] 

P(V2erfinv((l + 6^^)(2i— ^-l))<he,) < e'^^" 
P(-\/2erfinv((l + e^)(2^^^^-l))<h„_e3) < e"^'"'^'". (43) 

where ej^'^^ > 0, e-^^"^ > 0, and e^"'"^ > are arbitrarily small constants and €2^^^ £2^^' ^'^^ ^2"^^ ^^ 

constants dependent on e^ '^^ » ^i "^^ » and e^^ ""'^^ , respectively, but independent of n (essentially one only needs 
the direction of inequalities as in (43); however, a similar reverse inequalities hold as well). 

At this point we have all the necessary ingredients to determine ci, 62, and 63 and consequently Udep, 
II W(iep||2> and Cprim ('^' S' ^^ ^mag , T^socp)- We of coursc recall that in a random setup that we consider quan- 
tities cl, C2, C3, Vdep, llwrfeplb, and CprimV' g' h, Xmag , r^ocp) Can not really be determined. Instead what 
we will be determining are their concentrating points. The following theorem then provides a systematic 
way of doing so. 

Theorem 2. Assume the setup of Theorem 1. Let the nonzero components ofx have magnitude Xmag cmd 
let h be as defined in (14). Further, let rsocp = linin-^cxD -^7=^ <^nd Xmag = lini„^oo '"^"p? . Also, let Udepy 

W^deph, and Cpr!m(^' g' h' Xmag, rsocp) be as defined in and right after (16). Let a = ^ and {3^ = ^ be 
fixed. Consider the following 

?i->oo n \ Vvr I \ \ 2tt / 

1 -(erfinvi2^^-i))^ 



+ |/?«,a/— e 



Ed, 



dep 



/3. (J^i^^^^Wy 



n->oo n V27r I ^{erfinv(y^)Y 

/J. ( V2erfinv{2l^-l) \ _ p^ ( V2erfinv{2'-^ - l) \ 

^ V2^ \ ^ierfinvi2l^-W J^ ' ^ '"" ^ V2^ [ ^(er>v(2/^-i))^ ) + '' 



^(^-^3)^„l^m ^^.S-4f) (L/Je-(^'^-(^^ - L/Je-(^'^-(^^-)) 
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^(^1,^2,^3) = lim 



Eade 



^a^ + {l-e^-e2){xt%?{a - 0(91,92, ^3)) 



n^oo ^ R{92,93 



nm a a^ 1- E^<^^P ^Ja^ + {l-9,-92){xt%YS{9„92,9, 

-Difc'i,c72,c'3J = iim 



n^oo ^ R{92,9^) 

F{9i) = V2erfinv[ 



1-^1 

■l-/3» 



0(^2) = \^er>v(2i— ^ - 1) 

H{9s) = V2erfinv{2^—^-l). (44) 

Pui 

Set 

-{A{9i,92, 9^)B{9i,92, 9^) - S{9i,92, ^3)) 



N{9i,i 



'2,(73; 



A(^l,^2,^3)'-a + i^(^l,^2,^3) 



^{A{9i,92,9^)B{e^,92,9^)- S{9i,e2,9z)Y - {B{9i,92,9^Y + 9i+92 + e^-l){A{9i,92,9^Y -a + D{9i,92,9^)) 

A{9i,92,9^y-a + D{9i,92,9^) 

Let the triplet (9i, 92, 9^) be the solution of the following three equations 



„(^c) 



N{9^,92MG{92) + 



x'^Jg^N{9i,92,9^Y{a'^ - D{9i,92,9^)) + 2N{9i,92,9^)S{9i,92,9^) - {9^ + 92 + 9^-1) ^ ^ 

M(n a a^um^ xi?,%y/N{9i,92,9s)^a^ - D{9i,92,9^)) +2N{9i,92,9^)S{9i,92,9^) - {9i + 92 + 9^ - 1) 

^ a^ + {I - 9^ - 92){xtl? 

F{9i)N{9i,92,9^) = 1. 
(45) 

Then the concentrating points of Udf^p, \\^dep\\2> '^'^d ^l^J^^ {a, g,h,x mag, ^soc-p) can be determined as 

Evdep = N {9^92,9^) 

\]n{9i,92, 9^Y{a{l -93- ^2)^4^^ + D{9i,92, ^3)) - 2N{9i,92, ^3)^(^1, ^2, ^3) + (^1 + O2 + 9^ - I) 

E\\wdep\\2 = cr- 1 

Ni9u92, 9s)^ia - D{9i,92, 9s)) + 2N{9i,92, 9s)S{9i,92,9s) - {K + 92 + 9^-1) 



lim ■= = a 



^42™(^,g>h,x^,3,r,o,p) V^(^i' ^2, ^3)2(a- 15(^1,^2, ^3)) + 2iV(^i, 02,^3)5(^1, ^2, ^3) -(^1 + ^2 + 03-1) 



.A+(i_^-_^-3)(4|)! 



N{9^,92,9s)r^:X (46) 



Proof. Follows from Theorem 1 based on the discussion presented above and a combination of (27), (32), 
(33), and (35). D 
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The results from the above theorem can be used to compute parameters of interest in our derivation for 
particular values of /3w, a, a, Xmag, and rsocp- In the following subsection we will present a collection of 
such results. 

2.2.1 Theoretical predictions 



In this subsection we present the theoretical predictions one can get based on the result of the previous 
sections. We will split the presentation of the results into several parts. 



1) 



EW-WgepW 



Sllw, 



socpW 2 



as a function of x mag 



To present this portion (as well as several others that will follow) of theoretical results we will look 
at three regimes: 1) low a-, medium a-, and high a-regime. For each of the regimes we will show the 



theoretical results for 
of the low a-regime, a 



Eilw 



dep II 2 



^\\^^°-p\\^ as a function of x^- We will take a 



0.3 as a representative 

0.7 as a representative 

For each of the a-regimes we will look 



0.5 as a representative of the medium a-regime, and a 
of the high a-regime. We will consider rgocp = fsS^p = ^ J-^^ 
at two different sub-regimes: low /3^- and high /3^ -regime (which based on results from [53] is equivalent 
to low p- and high p-regimes). For each of these two sub-regimes /3^ will be selected based on the curves 
obtained in [53] (or those obtained in [52]) in the following way. In the low /3^ sub-regime we will set p = 2 



and r 



socp 



Jopt) 
I socp 



a 



whereas in the high /3^ sub-regime we will set p = 3 and r 



I socp 



(7-1 



10 • 



At the same time from [53] we will have r, 



socp 



Sopt) 
socp 



a 



VI 



a 



a,, 



socp 

n where a„, and /3, 



are 



such that (9) holds (we also recall on [53] where it was reasoned that the low /? regime is selected so that the 
pair {a, /3w) is well below the fundamental characterization (9) whereas the high /3 regime is selected so that 
the pair (a, /3^) is closer to the fundamental characterization (9)). The values for ''^'' = »^''°<'p»^ 
one can then get through the results of Theorem 2 for such (a, (3^) pairs, rg'^p are shown in Figure 3 as 



functions of Xmag- As can be seen from Figure 3, the values of 



EWwdeph 



-Bllw, 



socp II 2 



converge to p as 




4 




Ell »*pll2'<'=EI|w^JI,/a as a function of x!^^^^ 








3.5 




— p=2 

— p.3 




3 






^ 


§" 2.5 


y^ 


2 


/^ 


t 1-5 


/^ 


0.5 


^ 


° 








10 15 20 2 



E|| w^ |L/o=E|| w |I/ct as a function of x*^'^* 

" dep"2 " socp"2 mag 
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— p.3 
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Figure 3: ^^^^ = ^^^^^ as a function of x^; rsocp 
right — a = 0.7 



T^; left — a = 0.3, center — a = 0.5, 



Xmag incrcascs. This is of course in agreement with [53] where it was demonstrated that for rl^cp one 
has p = ^''^^""^''^ with overwhelming probability. Another interesting observation one can make is that the 
convergence is "faster" (or happens for smaller Xmag) for larger a. 
2)^ = ^4''^'("'g;';^"'"°^''^--'') as a function of x^^}g 



\/n 



y/n 



Similarly to what was discussed above (and is related to Hvi^socplb and ||w^^ep||2) one can determine 
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:-{dep) 



.(^c) 



the concentrating points of fobj and Crjm ^1^° ^^ functions of Xmag- To present these results we restrict 
ourselves to the medium a-regime, or in other words to a = 0.5. As above we again choose rsocp = 



' socp 



a 



j^^ and consider low p = 2- and high p = 3- regime. The obtained results are shown in 



Figure 4. As can be seen from Figure 4 y--' is larger for larger p. 



y/n 



J/2 



i(dep) 



1/2 



Ef , ./n "^=£^^"^^"'(0, g, h, X , r )/n "^ as a function of x 



mag socp' 




((SC) 

mag 



P=2 
p=3 



x(==)=x /n"^ 



15 



20 



Figure 4: 



Efobj 



^'iprim\'^'S:'^>^mag,i socp 



as a function of Xmac?; r^ocp 



i+p- 



^; a = 0.5 



3^ ^llw^eplb ^ g||w^o.p||2 ^^ a function of xt%; varying rsocp 



Another interesting set of results relates to possible variations in the Tgocp that can be used in (4). The 
results that we presented above assume an optimal choice for rsocp (in a sense defined in [53]). Namely, they 
assume that for a fixed pair (a, /3^) one chooses rsocp 



Sopt) 



' socp — (y\/{pL — aw)n where a^ and /3^ are such 
that (9) holds. In the worst-case scenario (or in the generic scenario as we referred to it in [53]) one has that 
choice rs^cp offers the minimal norm-2 of the error vector. However, such a scenario assumes particular x's 
which leaves a possibility that for a wide range of other x's the performance of the SOCP from (4) in the £2 
norm of the error vector sense can be more favorable. Of course as shown in Figure (3) this indeed happens 
to be the case. On the other hand that also leaves an option that one can possibly choose a different rgocp 
and get say a smaller norm-2 of the error vector for various different x. Below we present a few results in 
this direction. 

,(opt) 



r 



socp > 



We will consider again only the medium or a = 0.5 regime. For two different values of p, rsocp 
and /3^ we presented the results for ''^^ = »'^^°''P"^ jn Figure 3. In addition to that we now in Figure 5 
show similar results one can get through Theorem 2 for two different choices of rsocp- To be more precise, for 
yO = 2 we choose the same a and /3^ as in Figure and only vary rsocp over {o"\/0.05an, (T\/0.2ancr\/0.6an}. 
Clearly, choice a\/0.05an is smaller than 
rsS:p = cr\^0.2an. On the other hand for p 
in Figure 3 and vary rsocp but this time over {a\/0.05an, ay/O.lanaVO-^an}. Again, clearly, choice 



"^socp = a\/0.2an whereas choice a\/0.6an is larger than 
= 3 we choose the same a and /S^ as we have chose for p = 3 
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cr\/0.05an is smaller than 



socp 



aVO^Tan whereas choice a\/0.6an is larger than 



Sopt) 
socp 



a\/Th2an. 



It is rather obvious but we mention for the completeness that the middle rsocp choices for both, p = 2 and 
p = 3, cases correspond to the center plot in Figure 3. We make two interesting observations related to the 



E||w, |L/0=E|| w II /o as a function of x*^'^'; different r 

'' dep"2 " socp"2 mag socp 




E|| w^ |L/o=E|| w |L/a as a function of x*^'^' ; different r 

'' dep"2 " socp"2 mag socp 




-■■ r =(0.05m)" 

— '■socp=(0-'"") (P=3) 



30 



x'^^'^x /n" 



E\\wd^p\\2 



E\\v/socp\\ 



Figure 5: 

{aVO.Oban, a\^0.2anaV0.6an}; right 



„(«c) 



- as a function of Xmag for different rsocp', left 



2, r 



socp 



p = 3, rsocp G {fjVO.OSan, a\/0.1an(j\/0.5an} 

2StS t 

"'^^'"^p"^ could be larger than the one that can be obtained for '^^"^ ' 



,(op*) 



results presented in Figure 5. The first one is that Figure 5 suggests that if rgocp is smaller than rlocp then 

This actually happens to be the case. 



' socp 



A reasoning similar to the one presented in Section 2.4.2 in [53] can show that this is indeed true. Moreover, 
not only is it true for the x considered in Theorem 2 but it is actually true for any x. We skip the details of 
this simple exercise, though. The second observation is that if rsocp is larger than rs^J then for certain x 
(but of course not for all of them and certainly not for the worst-case or the generic one) "'^''"'^p"'^ could be 
smaller than the one that can be obtained for rs°Jcp ■ This of course suggests that a choice of rsocp larger than 
rso^p could be more favorable in certain applications and for a particular measure of performance. However, 
if one has no a priori available knowledge about x then adapting rgocp beyond rgJcp would be hard. 



(.{dep). 



Similarly to what was done above in part 2) one can also determine the theoretical predictions for 



socp 



'^ J Ob J ^prtm 



y/n 



\/n 



for a varying rsocp- As in parts 2) and 3) above, we restrict our attention only to the medium 



a = 0.5 regime. We also assume exactly the same scenarios as in part 3). The obtained results are shown in 

771 J" 

Figure 6. As in part 2) Figure 6 shows that "^^ is larger for larger p. On the other hand it also shows that 



"p-^ decreases as Vsocp 



\/n 



increases. This also follows rather trivially by the use of arguments from Section 
2.4.2. We skip this easy exercise as well. 

We conducted massive numerical experiments and found that the results one can get through them are 
in a firm agreement (as they should be) with what the presented theory predicts. In the next subsection we 
present a sample of the results obtained through the conducted numerical experiments. 
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Ef ^./n^'^=E£<'^^P'(CT, g- h, X , r )/n^'^ as a function of x'^'^' 

jjbj 'prim * ' =' mag socp' mag 




-'■ r =(0.05m)''^ 

socp 



6 8 10 

mag mag 



Ef /n^'^=E^<''®P'{a, g, h, x , r )/n"^ as a function of x'^'^' 



mag socp' 




10 

mag mag 



c(dep) 



Figure 6: -^ = """^ "''^'^'^"'"^'''"""^ as a function of xWg for different r^ocp; left — p = 2, r^ocp S 



\/n 



{crVO.OSan, (T\/0.2ancr-v/0.6an}; right — p = 3, r^ocp G {o"\/0.05an, cr\/0.1ano"\/0.5an} 



2.2.2 Numerical experiments 

Similarly to what was done in the previous subsection, we will split the presentation of the numerical results 
in several parts. The numerical results that we will present below are obtained by running the SOCP from 
(4). To demonstrate the precision of our technique we will in parallel show the results obtained by running 
(16). To make scaling simpler in all our numerical experiments we set a = 1. 

^ (7 (7 



as functions of Xmag 



In this part we will show the numerical results that correspond to the theoretical ones given in part 
1) in the previous subsection. To shorten a bit the exposition we will restrict our attention again only on 
the medium or a = 0.5 regime. We then set all other parameters as in the center plot of Figure 3 (these 
parameters are of course different depending if we are considering p = 2 or p = 3; below we will consider 
both of them). 

a) Low {a, (3w) regime, p = 2 

We first consider the p = 2 scenario. As mentioned above in our experiments we set a = 0.5, Tgocp = 

2 

Y^-2-Q. We then ran 
In parallel we ran (16) for the exact same parameters with 



1+P^ 



V0.2an, and (as shown in [53]) (3^ such that (a^t,, /3^t,) satisfy (9) and a^ 

(sc) 
mag- 



and 



(4) 300 times with n = 800 for various x 

only one difference; namely we ran (16) with n = 2000. The obtained results for 

are shown on the left-hand and right-hand side of Figure 7, respectively (given our assumption that a = 1 

''"^ and "^""'"P"^ are of course just -EHwdeplb and -E'||wsocp||2^ respectively). We also show in Figure 
7 the corresponding theoretical predictions obtained in the previous subsection. 

b) High {a, j3w) regime, p = 3 

We also conducted a set of experiments in the so-called "high" (a, /3^) regime. We used exactly the same 

Vsocp = VO.lan 



parameters as in low (a, /3^) except that we changed p from 2 to 3. Consequently we chose r^ 



and /3^ such that (oto, /3to) satisfy (9) and a 
ran (4) with n = 800 and (16) with n 



-r^-^a. As above we ran 300 times each (4) and (16). We 
2000. The numerical results obtained for p = 3 together with the 



theoretical predictions are again shown in Figure 7. From Figure 7 we observe a solid agreement between 
the theoretical predictions and the results obtained through numerical experiments. 
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Figure 7 

p = 3, r, 



Experimental results for \\^ ^°<'i'\\-^ and ''''^ as a function of Xmag', /> = 2, Vsoc-p = \/0.2an; 
■,ocv = \/(nOTl; left — SOCP from (4), right — (16) 



2) ^ and 






as functions of x 



(sc) 
mag 



In this part we will show the numerical results that correspond to the theoretical ones given in part 2) in 
the previous subsection. We then set all parameters as in Figure 3 (these parameters are exactly the same as 
in experiments whose results we just presented above). Of course we again distinguish two cases: p = 2 and 
p = 3. For both p = 2 and p = 3 we ran 300 times each, (4) and (16) and again we ran (4) with n = 800 



and (16) with n = 2000. The numerical results that we obtained for 7^-' and 



-^?pi-im (."^'Sihi^maai^socp) 



\/n 



are 



shown in Figure 8. We again observe a solid agreement between the theoretical predictions and the results 
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VO.lan; left — SOCP from (4); right — (16) 



obtained through numerical experiments. 
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3j ^ll^rf^plla and ^""^""^^"^ 



as functions of x mag; varying Vgocp 



In this part we will show the numerical results that correspond to the theoretical ones given in part 3) in 
the previous subsection. These results relate to possible variations in the rgocp that can be used in (4). We 
then set all other parameters as in Figure 5 (these parameters are of course again different depending if we 
are considering p = 2 or p = 3). 

a) Low (a, /3^) regime, p = 2 

We first consider the p = 2 scenario. As in part 1) of this subsection we set a = 0.5 and choose (5yj as 
in part 1). However, differently from part 1) we now consider two different possibilities for Vsocp, namely 

\/0.6an. We then ran (4) 300 times with n 



' socp 



\/0.05an and 



socp 



800 for various Xmag- 



In 



parallel we ran (16) with n = 2000. The obtained results for ^"'^;°"^"' and ^"'^^"^"' are shown on the 

left-hand and right-hand side of Figure 9, respectively (again, given our assumption that a 

and 



1 



-g^llWdeplh 



Bllw 



socpll 2 



are of course just -E^Hw^eplb and -EHw^ocplb. respectively). We also show in Figure 9 the 
corresponding theoretical predictions obtained in the previous subsection. 
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and ''"^ as a function of Xmaq', p = 2; rgocp £ 



{VO.Oban, V0.6an}; left — SOCP from (4), right — (16) 



b) High {a, P^) regime, p = 3 

We also consider the p = 3 scenario. As above, we set a = 0.5 and choose /3^ as in part 1) of this 
subsection. Everything else remain the same as in p = 2 case except the way we vary rsocp- This time 
we consider (as in part 3) of the previous section when p = 3 case was considered) rsocp = \/0.05an 
and rsocp = V0.5an. As usual (4) was run 300 times with n = 800 for various Xmag- In parallel we 



ran (16) with n 



2000. The obtained numerical results for V-^ 



and ^4''^'("'g''^"'"°«''^--^) as well as 
V" 



the corresponding theoretical predictions obtained in the previous subsection are shown on the left-hand 
and right-hand side of Figure 10, respectively. We again observe a solid agreement between the theoretical 
predictions and the results obtained through numerical experiments. Small glitches that happen in large 

(sc) 

Xmag regime could have been fixed by choosing a larger n. We purposely chose a smaller n to show that 
results are fairly good even when n is not very large. In fact, even a smaller n than the one we have chosen 
would work quite fine. 



y-i^(dep) /I \ 

•/n •/n 



as functions of Xmdg', varying rsocp 
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3; r, 



socp 



In this part we will show the numerical results that correspond to the theoretical ones given in part 4) 
in the previous subsection. These results relate to behavior of -J^ and — ^^^^^^ — , , mag, socp ^^^^ ^^^ 
varies rgocp in (4)- We again consider p = 2 or p = 3. 

a) Low (a, /3^) regime, p = 2 

The setup that we consider is exactly the same as the one considered in part 3a) of this subsection. We 
set a = 0.5, choose /3^ as in part 1), and considered two different possibilities for r^ocp, namely Tgocp = 

VO.OSan and rgocp = \/0.6an. The obtained results for y^^ and — ^^^^^^^^ — , . mag, aoep ^^ shown on the 
left-hand and right-hand side of Figure 1 1 , respectively. The corresponding theoretical predictions obtained 
in the previous subsection are also shown in Figure 11. 

b) High (a, /3^) regime, p = 3 

The setup that we consider is exactly the same as the one considered in part 3b) of this subsection. 
We set a = 0.5, choose /3^ as in part 1), and considered two different possibilities for Vsocp, namely 
V0.05an and rsocv = VO.ban. The obtained results for ^'1'^°°"^"" and ^""^""p"' are shown on the 



' socp 



left-hand and right-hand side of Figure 12, respectively. The corresponding theoretical predictions obtained 
in the previous subsection are also shown in Figure 12. We again observe a solid agreement between the 
theoretical predictions and the results obtained through numerical experiments. 

3 SOCP's problem dependent performance - signed x 

In this section we show how the SOCP's problem dependent performance analysis developed in the previous 
section can be specialized to the case when signals are a priori known to have nonzero components of certain 
sign. 

3.1 Basic properties of the SOCP's framework 

All major assumptions stated at the beginning of the previous section will continue to hold in this section as 
well; namely, we will continue to consider matrices A with i.i.d. standard normal random variables; elements 
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X. 



M. 



mag 



; P = 2; r, 



socp 



of V will again be i.i.d. Gaussian random variables with zero mean and variance a. The main difference, 
though, comes in the definition of x. We will in this section assume that x is the original x in (3) that we are 
trying to recover and that it is any fc-sparse vector with a given fixed location of its nonzero elements and 
with a priori known signs of its elements. Given the statistical context, it will be fairly easy to see later on 
that everything that we will present in this section will be irrelevant with respect to what particular location 
and what particular combination of signs of nonzero elements are chosen. We therefore for the simplicity of 
the exposition and without loss of generality assume that the components xi, X2, . . . , ^n-k of x are equal 
to zero and that the remaining components of x, x„_fc_|_i, x„_fc+2, . . . , x„, are greater than or equal to zero. 
However, differently from what was assumed in the previous section, we now assume that this information 
is a priori known. That essentially means that this information is also known to the solving algorithm. Then 
instead of (4) one can consider its a better ("signed") version 



mm 

X 



subject to 



l|x||i 

||y - Ax.\\2 < rsocp+ 
Xj > 0, 1 < i < n. 



(47) 



Also, one should again note that rsocp+ in (47) is a pai^ameter that critically impacts the outcome of any 
SOCP type of algorithm (again, for different rsocp+'^ one will have different SOCP's). The analysis that we 
will present assumes a general rsocp+- However, we do mention right here that problem (47) is not feasible 
for all choices of x, a, /3+, a, and rsocp+- Unless mentioned otherwise what we present below assumes 
that X, a, /3^, a, and rsocp+ are such that (47) is feasible with overwhelming probability. For example, as 
discussed in [53], a statistical choice rsocp+ > (y\fm guarantees feasibility with overwhelming probability. 
Of course, there are other choices of parameters x, a, /3^, a, and Vsocp^ that guarantee feasibility as well. 
Towards the end of this section we will mention some of them and address this general question of feasibility 
in more detail. 

Given the positivity of Xj , 1 < i < n, one can, of course, replace £1 norm in the objective of (47) by 
the sum of all elements of x. However, to maintain visual similarity between what we will present in this 
section and what we presented in Section 2 we will keep the i\ norm in the objective. Along the same lines, 
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in what follows we will try to mimic the procedure presented in the previous section as much as possible. 
On such a path we will skip all the obvious parallels and emphasize the points that are different. 

As a first step in making the presentation of the "signed" case as parallel as possible to the presentation 
of the "general" case we will again introduce a few definitions that will turn out to be helpful in what follows. 
First, let us define the optimal value of a slightly changed objective from (47) in the following way 



fobj+ = min 

X 

subject to 



X 1 



mi 



||y-^x||2 < rsocp+ 
Xj > OA < i < n. 



(48) 



As in the previous section, fobj+ is clearly a function of a, x, A, v, and rsocp+- To make writing easier we 
will adopt the same convention as in Section 2 and omit them. Also let :x.socp+ be the solution of (47) (or the 
solution of (48)) and let ■Wsocp+ S R^ be the so-called error vector defined in the following way 



w< 



ocp^ 



X. 



ocp-\ 



X. 



socp^ 



(49) 



and 



As in Section 2 our main goal in this section will be to provide various characterizations of w 

fobj+- Throughout the paper we will heavily rely on the following theorem from [53] that provides a general 

characterization of ■Wsocp+ and fobj+- 

Theorem 3. ( [53] — SOCP 's performance characterization; signed x) Let w be an n ^ 1 vector of i.i.d. 
zero-mean variance a'^ Gaussian random variables and let Abe an m x n matrix of i.i.d. standard normal 
random variables. Further, let g and hbe m x 1 and n x 1 vectors of i.i.d. standard normals, respectively 
and let z be n X 1 vector of all ones. Consider a k-sparse x defined in (6) and a y defined in (3) for 
X = X. Let the solution of (47) be Xsocp+ cind let the so-called error vector of the SOCP from (47) be 
•Wsocp+ = ^socp+ — X. Let rsocp+ in (47) be a positive scalar Let n be large and let constants o. = ^ and 
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Pw ~ n ^^ below the following so-called signed fundamental characterization of£i optimization 

fY ~{erfinv{2^-^-i)Y 
(1 - Pl)^^ + V2erfinv{2—^ - 1) = 0. (50) 

Furthermore, let x, a, /3^, a, and rsocp+ be such that (47) is feasible with overwhelming probability and 
ES,prim+if^, S; h, X, rsocp+) defined below is finite. Consider the following optimization problem: 



iprim+{cr, g, h, X, rsocp+) = max a\ WgWlu"^ - \\vh + z - A||^ - V] AjXj - i'rsocp+ 

1/ \ V ^— ' 



i=n— fc+1 



subject to V >Q 

Ai>0,l<z<n. (51) 

Let v^ and A+ be the solution of (51). Set 

iin^ii ||^h + Z-A+||2 ,^,, 

||w+||2 = cj I ^^ ^ ^ ■ (52) 

Yl|g|||z^+ - ||i/+h + z- A+lll 
Then: 

P(||x + w^ocp+lli - ||x||i G (^Cprim+(o-,g, h,x,rsocp+)) - ei'*°''^^|£;Cprim+(o-,g,h,x,rsocp+))|, 

-K?prim+(o-,g, h,X,rsocp+)) + ef |-ECprim+(0",g, h,x,rsocp+))l) = l-e"^2 " (53) 
and 



P((l - 6i^°^^))i?||^^||2 < \Wsocp-,h < (1 + ei^°'=^))i^||^||2) = 1 - e-4^°"^", (54) 

where ^^^"'^'''> > /i' a« arbitrarily small constant and e^°'^'^' 1$ a constant dependent on ^^^"'^'^> and a but 
independent of n. 

Proof. Presented in [53]. D 

Remark: A pair (a,/3+) lies below the signed fundamental characterization (50) if a > a+ and q+ and 
/3+ are such that (50) holds. 

3.2 Problem dependent properties of the framework 

To facilitate the exposition that will follow we similarly to what was done in Section 2 (and earlier in 

[53,54,57]) set 

n — l"(l)i"(2)'- • • '"(n-fc)'"n-A:+l'"n-fc+2'--- '"n J ) K-^-^J 

where [hU, hL|, . . . , h^^_^|] are the elements of [—hi, — h2, . . . , — h„_fc] sorted in increasing order and 

[^iifc+i' K^fc+2' • • • ' hn ] are the elements of [-h„_fe+i, -hn-k+2, ■■■ , -h„] sorted in decreasing order 
(possible ties in the sorting processes are of course broken arbitrarily). One can then rewrite the optimization 
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problem from (51) in the following way 

I = " 

u,X V ■^ — ' 

i=n—k+l 

subject to z/ > 

Ai > 0, 1 < i < n. (56) 

In what follows we will restrict our attention to a specific class of unknown vectors x. Namely, we will 
consider vectors x that have amplitude of the nonzero components equal to Xmag- In the noiseless case these 
problem instances are typically the hardest to solve (at least as long as one uses the signed version of the £i 
optimization from (2)). We will again emphasize that the fact that magnitudes of the nonzero elements of x 
are Xmag is not known a priori and can not be used in the solving algorithm (i.e. one can not add constraints 
that would exploit this knowledge in optimization problem (47)). It is just that we will consider how the 
SOCP from (47) behaves when used to solve problem instances generated by such an x. Also, for such an 
X (56) can be rewritten in the following way 



n 



^prTm+{^^^^'^^^niag,rsocp+) = max (T^ ||g||^I/2 _ \\y\y\- _ z + A||| - X^ag ^ Aj - Vrsocp+ 

i=n—k+l 

subject to V >Q 

Ai > 0, 1 < i < n. (57) 

Now, let i'dep+ and X^'^^p^') be the solution of (57). Then analogously to (52) we can set 

||Wrfep+||2 = O- , _ (58) 



In what follows we will determine ||wrfep+||2 and ^^^^^{(7,^,h.,Xmag,'rsocp+) or more precisely their 
concentrating points ^||wrfep+||2 and £^^^^^^^^(o-,g,h,a;maa,?^socp+)- All other parameters such as Vdep+, 
^(dep+j ^^^ ^^j^j some of them will) be computed through the framework as well. 

We proceed by following the line of thought presented in Section 2 and earlier in [53,57]. Since A^'^'^^'^) 
is the solution of (57) there will be parameters c^ and c^ such that 

X{dep+) = [^('ieP+) ^ ^{'ieP+) ^ . . . ^ yidep+) ^ 0, 0, . . . , 0, A^'^T^ , A^?^+^ , . . . , A(f ^P+)] 

and obviously cf < n — k and n — k < c^ < n. At this point let us assume that these parameters are known 
and fixed. Then following [53,57] as well as what was presented in Section 2 the optimization problem from 
(57) can be rewritten in the following way 






^prim+ (O-, g, h, X, rsocp+) = ^ max '^^J\\S\\2^ - \Wi\^+^^.^- Zc+ + l:n + ^ct + l-.nM - ^rnag 2^ Xi - iyrsocp+ 

subject to u > 

Xi>0,c^ + 1 <i <n 

Xi = 0,cf + l<i<4. (59) 
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To make writing of what will follow somewhat easier we set 



i=n—k+l 

Similarly to what was done in Section 2 we then proceed by solving the optimization in (59) over u and 
'^c++in- "'"° ^° ^° *^ ^^^^ ^°°^ ^^ ^^^ derivatives with respect to Aj, c^ + 1 < z < n, of the objective in (59). 
Computing the derivatives and equalling them to zero gives 

-± = 0,ct + l<i<n 

dXi 

-(z/h+-Zj + Ai) + , ^ ^ . ^ 

^ ., „^ „ „ ,-, , ^=7^ - Xmag = 0, €3 +l<l<n 

I&II2 II c+ + l;n cj+l-.n Cj^+l:?ill2 



^^ Aj Zj -h Z^n. - ^ yl|g|l2'^ II c+ + l;n ^c+ + l;n + ^c+ + l;n lb' ^2 + ^ ^ * ^ ^ 

^^ A, = -^^||g||iz^2 _ ||zyh++^^^^ - z^+^i^„ + A^++i^„||2 + Zi - i/h+, c+ + 1 < i < n. 

(61) 

From the second to last line in the above equation one then has 

(A,-— z,'+z^&^) = "^o^ (llglUt^ — ||i/h"h_ , — z + , -, ++A + , 1 +||o — llz^h + , -, — z + , -I +A + , , lU) 

V ' ' I ' _2 Vlloll^ II c +l:c c^+liCj Cj +1X2 112 II cj+l:n cj+l:n ' cj+l:nii^' 

and after an easy algebraic transformation 

•^mag /n 112 2 



(A. - z, + ^h+)^ = . , '""^+. , ihUu^ - \Wh^,^,.^, - z^.^,^^.||^). (62) 

(T^ -I- ir). — rJ Vt^ c, +i.c, 1 T^ 2 

Using (62) we further have 



a^ + (n- C2 jX^ag ''i ^'^''^ 



(T 



I 1 1 / 9 1 1 1 + I \ M V / 1 1 1 1 / 9 1 1 1 + 119 

e ^z/^ — U^h^ — z + , -, + A + I -I = — — 1 / P. t'^ — '^" + + — z + I -I +9. 

'^+^^" c,+l:n^ c,+l;nll2 / 2 + (^ _ c+)a;2^^ V "^"^ " 4+1:4 c, +l:c, \\2 

(63) 
Plugging the value for Aj from (59) in (60) gives 

n 

^^°^^^^ = ^^llslli'^^ - W'^Kt+l-.n ~ ^4 + l:n + K+ + l:n\\2 " ^mag J] ^i " ^^^ocp+ 

i=n—k+l 

a"^ + (n- ct)x^ 



2 /•^mag /7, rTo 9 n 1+ ^ ii2 

^ Vllslli'^' - \\^Kt+l:n - V+1- + V+I-II2 

n 

j=c++l 
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Combining (63) and (64) we finally obtain the following "signed" analogue to (25) 



n 



^{obj+) ^ ^^2 + (n - 4)x'^raag J WSWW " \\^K++l:c+ ~ ^c++l:c+\\2-^i'^socp+-Xmag Yl K)-Xmag{n-4). 

1 ' 2 

(65) 
Equalling the derivative of ^(°^-?+) with respect to v to zero further gives 

d^{obj+) ^ 

du 

ysocp+ XfYiag y ^ "-i ) — ^^' 



a^ + {n-4)xl^^g) Y||g||iz^2 _ ||jyh+^^^^^^ _ z^+^^^^+||2 



(66) 



Let 



Sdep+ — (h + 



\T^ 



Z, 



c+ + l:c+' -^c^ + lxj 

ddep+ = Y. ^^t? 
i=ci+l 

n 

fdep+ — fsocp+ XfYiag / ^ -H-j 

i=C2 + l 

(^dep+ — 



Q / -I— \ O (J,G'0~\ 



(K+ +1:4^^^4+1:4 y(^^ + (n - 4)xl^agSdep+ 

bdep+ = — I = • (67) 

■J a + [n — C2 )X^ag fdep+ 

Then combining (66) and (67) one obtains 

{ttdep+y + hdep+f = ||g||il^^ - lkh+ + - V+l-c+lli- (68) 

After solving (68) over u we have 

-{adep+bdep+ - Sdep+) - J {adep+bdep+ " Sdep+)'^ - {b^^^j^ + ||Zc++i:c+ Il2)(<^dep+ ~ Ilgll2 + ddep+) 

V = ^^^ 2 ini2 — 'a ■ 

<^dep+ ~ Ilgll2 + ddep+ 

(69) 
Following what was done in Section 2 and earlier in [53, 57], we have that a combination of (61) and (69) 
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gives the following two equations that can be used to determine ci and C2. 

Xr 



r+ 1 -^rnag u ul o iir+ i\ ii2 n 

cj ^2 (T \ ii^"!!^ II c^+l:n Cj -|-i.n Cj^ -|-i.niiz 



h^ 



{adep+bdep+ - Sdep+) — J {adep+bdep+ — ^dep+Y — {bdep+ + INc++l:c+ Il2)('^dep+ ~ Ilsll2 + ddep+) 

^dep+ ~ Ilsll2 + ddep+ 

(70) 

The last term that appears on the right hand side of the first of the above equations can be further simplified 
based on (63) in the following way 



^V I|g|l2^^ - J^'^ddep+ + 2iySdep+ - (C2 " Ci 



a + [n C2 jx^^g 

where we of course recognized that ||z^+ , ]^.^+ II2 = c^ — c^. Combining (67) and (71) one can then simplify 
the equations from (70) in the following way 



Z/h++ - Z^+ + """^ yllglli'^^ - ^'^ddep+ + 2uSdep+ - (c^ - C^) 

yo- + (n - C2 )Xjnag 

_ -{cLdep+t>dep+ — Sdep+) " \J {(^dep+bdep+ — Sdep+) " (^dep+ + ("^2 ~ ^1 )){'^dep+ ~ \\^\\2 + ddep+) 

c+ 7^ iu7ii2 

''1 a — """ 



dep+ llSib + "'dep+ 

(72) 



Let cj" and Cg be the solution of (72). Then 



~(o^dep+Odep+ ~ Sdep+) — \J {adep+bdep+ ~ Sdep+) ~ \Pdep+ + (C2 — Ci)){adep+ — ||g|l2 + "dep4 

i^dep+ = .^2 TT^ T 

fldep+ — llgib + "(iep+ 

__ (73) 

where s^i^, ddep+, Od^^, and bdep+ are Sdep+, c^dep+, Odep+, and hdep+ from (67) computed with cl and 
C2. From (58) one then has 

II u+ I \ {dep+) II 

I II c++l:n c++l:n c++l:n 

|Wrfep+||2 = cr , (74) 

lo-l|2,/2 II,,, \.+ „_ I x(<^ep+) ||2 

Combining (71) and (74) one further has 



,, ,, ,MWdep+i^-4)^^ + ^lep+ddep+-2uS^+ + {4-4) 

W^dep+h = <^- , ^ ^ ^ ^= • (75) 

V llglli^ip+ - ^lep^ddep^ + 2j^dep+s'd^ - (4 " 4) 

Combination of (73) and (75) is conceptually enough to determine ||W(^ep+||2 (and then afterwards easily 

-^Wim+('^' g' ^^ Xmag,rsocp+))- The part that remains though is a computation of all unknown quantities 
that appear in (73) and (75). We will below show how that can be done. In doing so we as usual substantially 
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rely on what was shown in [53, 57] and assume a familiarity with the procedures presented there. 

The first thing to resolve is (72). Since all random quantities concentrate we will be dealing (as in 
[53,57]) with the expected values. To compute the solution of (72), ci and 62, we will need the following 
expected values 

(76) 
As in Section 2 we easily have 

EM\l = m. {11) 

Let cl = {1 — 6f)n and c^ = 02'n where Of and 6*^ are constants independent of n. Then as shown 
in [53,57] 

E\\h\,, .||2 , ^+ /^(erfinv(2i:^-l)) , 



j.^ _ c++l:n-fcll2 _l-/3+ 



n v/27r 



, (erfinv(2^-^-i))2 
\ e '--Pw 



+ ^r-/3^. (78) 



where we of course recall that 13^ = ^. Also, as in Section 2 and earher in [53, 57] we have 

^l|h+ ,^,.^+lli p+ /V2erfinv(2i-^-l) 

n ^2^ (erfinv{2^-i))2 

\ e f^-m 

Following further what was established in [53,57] we have 

,. ^^^Ki+V.n-f^ct+l:n-k) ( [T - (tV^m {2'-^ -1)A 

hm ^ = (l-/3j;)W— e i-zs^ 

n^oo n \ V 27r / 

^^^Ki-k+i-c+^'^'^n-k+i:4) ( ^ [T -{erfinv{2i^-i))2\ 

ra-5>oo n \ V 27r / 

From (80) we also have 

lin, - — ^^- - - = - ( /3:W-e ^i -' 1 . (81) 

n^oo n \ V 27r / 

The only other thing that we will need in order to be able to compute cf and c^ (besides the expectations 
from (76)) are the following inequalities related to the behavior of h^ and h"*^. Again, as shown in [53,57] 

P(V2erfinv((l + e^M(2f^-l))<h+) < e^^^^" 

h++ ^ - ft+ _ K+ 

P(V2erfinv((l + e/^)(2 ^ _ l)) < h++) < e''^ ". (82) 

&+ ii+ ii+ h+ 

where e^^"^ > and e^'^^ > are arbitrarily small constants and €2"^ and 63 '^^ are constants dependent on 
e^ ^ and e^ ^ , respectively, but independent of n. 



EiE7=c,+i^t) fo+ A^_-(erfinv(2i^-i))2 
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At this point we have all the necessary ingredients to determine cj" and C2 and consequently Vdep+, 

\Wdep+\\2, and ip^^_^{a, g, h, Xmag-,rsocp+), or to be more precise their concentrating points. The follow- 
ing theorem then provides a systematic way of doing so. 

Theorem 4. Assume the setup of Theorem 3. Let the nonzero components ofx have magnitude Xmag cmd let 
h+ be as defined in (55). Further, let r^^^j^ = lim„_>oo ^''°%'^ and Xmag = lini„_^oo ^"^pp . Also, let ydep+> 

||wrfep+||2, and Cprim+('^' S) h, Xmag':rsocp+) be as defined in and right after (57). Let a = ^ and fit = ^ 
be fixed. Consider the following 

^tn+ n+, r ^'<'-P+ (f^ «+^ n~ -ierfinv(2'^-i)f\ ( [T -ierfinv{2'-^-i)f\ 



Fd, , 1 - /?+ / V2{erfinv{2^^ - 1)) 
"^- ^ V2^ \ ^ierfinvi2'^-i)r ' 

0+ (^erfinv{2'-^-l) 

e P" 



Erdep+ (sc) . (,r) ( n+ I '^ -{erfinv {2^-^-1)) 



dep+^(sc) (sc) P+J± 



^(^2 ) = }Too -^ = '-'^P^ + -"^^ [M^' 



K 



A(t)T,0Z) = lim ^±— = — : 



Ehdep+ V^' + (1 - et){x^l%YS{et,e^) 



B{9i,en = lim ^ 

F{et) = V2erfinv{2^^ - I) 
G{e+) = V2erfinvi2^^-^-l). 

Pw 



(83) 



Set 



N{9+,et) 



-iAiet,e+)B{et,et)-s{et,et)) 
A{et,etr-a + D{9t,9t) 



{A{et, et)B{6t, et) - s{et, e+)y - {B{et, e+r + of + 9+ - i){A{et,etr -a + D{e+, e+)) 



A{et,9tr-a + Diet,et 
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Let the pair (O^, 62) be the solution of the following two equations 



xt}g^JN{et,etY{a-D{et,et)) + 2N{et,et)s{et,et)-{et + et-l) 

N{e^ , 6*2 )G(02 ) "^ / = 1 

a^ + {I - et){xtJ9f 

F{el)N{9+,e+) = 1. 

(84) 
Then the concentrating points ofvdep+, \\'^dep+\\2> cmd Cp^im+i'^^ S' '^' Xmag,rsocp+) can be determined as 

Eudep+ = N{et,e+) 

yjN{et. g+)2(«(l - g)4l^ + DJ^.oi)) - 'iNjOt, Ot)S{Ot, 6+) + {9+ + 9+ - 1) 

E\\wdep+\\2 = cr- , ^ ^ ^ ^ ^ ^ ^ ^ 

^N{9t, ^2^)2(a - D{9t, 9+)) + 2Ni9t,9+)Si9t, 9t) - (^^ + 9^ - 1) 



EClf,T2+i<',S,h,Xmag,rsocp+) ^N{9t,9t?{a - D{9i,9t)) + 2Ni9t ,9t)S{9t ,9+) - [9^ + 9^ - 1) 



iim — — = (J 



1+1 



'2)^7^ 



N{9t,et)r[':l^. (85) 



Proof. Follows from Theorem 3 based on the discussion presented above and a combination of (67), (72), 
(73), and (75). D 

The results from the above theorem can be used to compute parameters of interest in our derivation for 
particular values of /5+, a, a, Xmag, and rsocp+- In the following subsections we will present a collection of 
such results. However, before doing so in the next subsection we take a look back and discuss the feasibility 
of (47) in a bit more detail. 

3.2.1 Feasibility of (47) 

As we have mentioned at the beginning of this section the optimization problem in (47) is not necessarily 
feasible for all possible choices of A,w,a,yL, and rsocp+- Analogously, (51) and (57) are not necessarily 
bounded for all choices of a, /3^, cj, g, h, x, and rsocp+- Below we provide a brief discussion on potential 
unboundedness of (51) and (57). We will split the discussion into two parts. First we focus on a couple of 
scenarios where the objective of (51) and (57) is bounded. Afterwards we present a procedure that can be 
used to determine Xmag for which (57) becomes unbounded and (47) infeasible. 
1) Universally feasible scenarios 

Clearly, the most critical case for (57) to be unbounded is that Xmag = 0. Even if Xmag = one can 
distinguish two important scenarios for parameters a,/3^,a, and rsocp+ such that the objective in (57) is 
bounded with overwhelming probability. 

a) rsocp+ > CTy/m 

The first scenario assumes rsocp+ > cr\/m. Then for any combination of (a, /3+) that lies on or below 



31 



the signed fundamental characterization (50) and any a one has that 



o-yllglli^^ - ll^h+ -z + X\\l -vrsocp+ = ^{(^'^M\l - l|h+ - - + -\\l - r socp+) ■ (86) 

Let ef'^"''^' > be an arbitrarily small constant and let ^2"^"^ be a constant dependent on e^'^"'^' but 
independent of n. Since 

one has that with overwhelming probability 



i^(||g||2 < (1 + ei ')y/rn)>l-e 



^2 



g||^i/2 - ||z/h+ - z + A||^ - z/rsocp+ < 

which implies that the objective in (57) is indeed bounded with overwhelming probability. 

b)a< 0.5 

The second scenario that we consider assumes that a < 0.5. Then for any combination of (a, /3+) that 
lies on or below the signed fundamental characterization (50) and any a one again has that the objective in 
(57) is bounded with overwhelming probability. To show this we will look at the expression on the right 
hand side of (86) instead of looking at the objective of (57). Clearly, to have that expression unbounded one 
must have 1/ — )■ 00 and the term in the parenthesis must be positive. Also the term under the square root 
must be nonnegative. Since Zj = 1 for any i one easily has that z/u would have to converge to 0. Let us 
therefore look at the following function ({X) 

C(A) = ||g||i-||h+ + ^||2. (87) 

Furthermore let 

C = maxC(A) = max||g||^- ||h+ + -||i (88) 

A A 1/ 

Then it is not that hard to see that 

C=l|g|li-||h5.^,^,.lli, (89) 

for certain c{ + 1 < n — k and n — k + I > C2 < n. For any arbitrarily small constants e^'^"'^' > and 
eg > and constants e)^ , eg dependent on Cg and e^ , respectively but independent of 

n one easily has 



and 



. (feas) 



^(h;,_4.e.)^_^>0)>l-e 
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One then with overwhelming probabihty has that c{+ < (1 + e^'^''""'^)^ and 4+ > (1 - ep"""^)^^. 
From (89) it then easily follows that with overwhelming probability 

C=||gi-||h+^_^^^^,Jli<m-^. (90) 

On the other hand if a < 0.5 then m < ^ and from (87), (89), and (90) one has that with overwhelming 
probability ({X) can not be positive for zv — > co. This in return implies that with overwhelming probability 
the objective in (57) is not unbounded. 

We also mention that the feasibility of (47) in the above mentioned scenarios could be deduced by 
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looking directly at (47). For example, if rsocp+ > o"m then x = x is with overwhelming probability feasible 
in (47). On the other hand if a < 0.5 then based on results of [58] (and earlier [23]) one has that the norm-2 
in the constraint of (47) can with overwhelming probability be made zero (this is in fact exactly the inverse 
problem of the one considered in [23, 58]). We will present this small exercise in one of our forthcoming 
papers since it fits better the topic there. 

Also, we should mention that one can define many other scenarios where (57) is bounded. However, we 
restricted only to the above two since they are relatively simple to describe and have a nice connection to 
already known results. 

2) Finding the feasibility breaking point 

In the rest of this subsection we will present a general mechanism that can be used to determine a 

(sc) 

critical Xmag for which the objective in (57) becomes unbounded and (47) infeasible. We start by rewriting 
the objective of (57) in the following way 



^(^V/Ilslli - l|h+ \--\\2-Xmag ^ —-rsocp+)- 

To have the above expression unbounded one needs Aj = i^A^ , u ^ oo, and the expressions under the 
square root and in the parenthesis to be positive. Let us then consider 

n 



max a^J\\g\\l-\\h+ + X^W^-Xmag ^ Af^-rsocp+ 

subject to Af ^ > 0, 1 < i < n. (91) 

Let A*^-^*^*^*^ be the solution of (57). Following the line of thought presented in Sections 2 and 3.2 there will 
be parameters cl "'^' and C2 such that 

\ifeas) _ \\ifeas) .(feas) Meas) „ „ „ Meas) Afeas) \ifeas)i 

A — [A^ , A2 , . . . , A ^feas) , U, U, . . . , U, A ^f^as) -. ) ^ (feas) „} • • • ) ^71 J 

and obviously c-[ < n — k and n — k < Cg < n. At this point let us assume that these parameters 

are known and fixed. Then following what was presented in Sections 2 and 3.2 the optimization problem 
from (91) can be rewritten in the following way 



/Ti m2 M-r -f- \ \\2 \ ^ \ 

max <^-y/l|g|l2 ~ ll'lc+ I x-n ~'~ c*^'""'' + l;nll2 ~ ^mag /^ ^i — rsocp+ 

\feas) , , V 1 ■ 1 ' -711 

Y '+l:n l=n — k+l 



^ (feas) 

subject to Aj > 0, C2 +1 < i < n 



A, = 0,cS^'"^)+l<i<4^''^'). (92) 

Based on the arguments just above (91) one has that if the optimal value of the objective in (91) is positive 

(sc) 

then the optimization problem in (57) is unbounded. We will then call the largest Xmag for which the 

(sc) 

optimal value of the objective in (91) is positive the feasibility breaking point. To determine such an Xmag 
we proceed by solving the above optimization problem. To make writing of what will follow easier we set 

n 



t(feas) _ /||o.||2 _ ||u+ i \ ,, ^ IP _ r \^ \- - r , ^93^ 

? — "-WII&II2 II c++l:n c*-'^^"°'+l:nll2 -''mag 2_^ ^i ' socp+, Ky-*) 



n-k+1 
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and 



^ifeas) ^ j^ax C 



ifeas) 



^ JJeas) 

'^1 



+ l:n 



subject to Aj > 0, 4 + 1 < i < n 

A, = 0,cp"') + l<i<4^""^\ (94) 



Similarly to what was done in Section 3.2 we then proceed by solving the optimization in (92) (or the one 
in (94)) over A (feas) . To do so we look at the derivatives with respect to A 
objective in (92). Computing the derivatives and equalling them to zero gives 

0, C2 ' + 1 < t <n 



in (94)) over A (feas) . To do so we look at the derivatives with respect to Aj, 4 + 1 < i < n, of the 
Comf 



dXi 



Irrll^ \\U^ _l_ \ l|2 

cY +l:n Cj^ -t-i.n 



\ _l_ V^+ — ™"g /llrrl|2 ||i;+ _i_ \ ||2 Ajecis -, / • ^ „ 



(95) 



From the second to line in the above equation one then has 



X 



2 



i' \ I u+^2 _ "i-<^9 ni ii2 IIU+ I \ ii2 iir , \ ||2\ 

(Ai+h. j _— 2-(||g||2- ||h^(/ea.)^^^^(/ea.)+A^(/ea.)^^^^pa»)||2-||h^(/ea.)^^^^ + A^(/ea.)^^^^y 

and after an easy algebraic transformation 



X? 



(A. + h+)2 = — "^rL... . (llglli-||hV..),. (/..Jl2)- (96) 






Using (96) we further have 



lrrl|2 _ 111,+ _|_ \ l|2 _ /llrrl|2 _ ||Vi + 

(97) 
Plugging the value for Aj from (96) in (93) gives 



a 

/ll^ll^ lll^T 

^(/eas)_j_-| .^(/eas) Il2' 

mag 






c(/eas) _ /||o-||2 _ ||h+ _l\,, , IP _ r \^ A 

0" + ("- - ^2 )Xmag /|| „||2 _ 111,+ , \ ||2 

8II2 II" (/eas) , -,.„ "T ^^(/e«s)^^.^||2 



^mag 



J2 h+ - r,o,p+. (98) 



(feas) , -, 
t=C2 +1 
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Combining (97) and (98) we finally obtain 



t(/eas) _ / 2 I f^ _ ^(/ea«)\ 2 / ||p-||2 _ ||h+ P - (r ,-T "^ h + ) r99^ 



J=C2 +1 



Following what was done in Sections 2 and 3.2 we have that a combination of (96) and (98) (together 
with the fact that the elements of A are nonnegative) gives the following two equations that can be used to 
determine c\ """^ and Cg 



u+ I -^mag n 112 ||t,+ 

a + [n C2 j^mag 



h>e..) < 0. 



(100) 



Let cl and Cg be the solution of (100). Then from (99) we have 



^(/ea.) = y ,2 + (, _ cr-))x^„ /||g||i - ||ht^ ^J2 - (..„,,^ - .„., J] h,+ ) 



(101) 
Combination of (100) and (101) is conceptually enough to determine the feasibility breaking point for Xmay 
The part that remains is a computation of all unknown quantities that appear in (100) and (101). To do it we 
will as usual substantially rely on what was shown in previous section and basically in [53,57]. 

The first thing to resolve is (100). Since all random quantities concentrate we will again deal with the 

expected values. To compute the solution of (100), c^ and Cg , we will need the following expected 
values 

n 

As in previous sections we easily have 

EM\l=m. (103) 

Let c^ = (1 — ^1 )n and C2 = 62 '^^n where Q)^^'^^' and ^2 ^^ constants independent of 



n. Then as shown in [53, 57] 



S||h+,„., \\l , n^ /^/2(erfinv(2 ^~^Cr' ^,, . 



lim — 



n V27r 






-I_fl{/eas) 

(erfinv{2— i-^ 1))2 

= 1— Pu) 



where we of course recall that /?+ = ;|. Also, as in Section 2 and earher in [53,57] we have 



-£^h , ,, (/eas) 2 «+ / \/2erfinv(2-^-^2+ 1) , ,, , 

lim "-^^-- = % ^1- I + ei^-^ - 1 + /5-. (105) 

^^ I (erfinv(2^-% 1^^2 



n^<» n ^/2^ I .„^„...„i-«2^_i))o I ^ 
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Following further what was established in [53,57] we also have 

^iT.'^^Meas) hf) I n~ -(erfinv(2 '~'\°°' -i))2 , 

lim '-f2 ±^ = - (3+J—e ^^ . (106) 

n-5>oo n \ \ 2-K 

The only other thing that we will need in order to be able to compute c^ and C2 (besides the 

expectations from (102)) are the following inequalities related to the behavior of h^/e„s) and h^^^^^j . Again, 

as shown in [53,57] 

-+ h+ 

^ (feas) -I _ aU^^as) _ ^(/etts) 

P(V2erfinv((l + 6/^ )(2 ^ ^ . - 1)) < &+,_)) < ^^^ 

-+ h+ 

^ (feas) 1 _ aU^as) _ ^(/eas) 

P(^/2erfinv((l + 6/^ )(2 -^ 1)) < &+,_)) < e^^ "• (107) 

J,feas) ^(feas) J^feas) Jfeas) 

where e^^ ^ > and e^ ^ > are arbitrarily small constants and €2 ^ and 63 ^ are constants 
ii+ h+ 

^(feas) ^(feas) 

dependent on e^ ^ and e^ ^ , respectively, but independent of n. Also, we find it useful for what 
follows to introduce the following definitions 



Sfeas = ^ hf 



4-f""''+l 



''feas 



(feas) 
'^2 










1. 


i^tf 








,-_^(/e«») 


+1 


n 






'f^socp+ 


~ Xmag 


1. 




^■ 




i= 


(feas) 
'^2 


+ 1 





ffeas — fsocp+ Xmag / ^ ^^j • vJ^'-'oJ 



At this point we have all the necessary ingredients to determine c^ ' and C2 and consequently ^(Z'^^'^), 
or to be more precise their concentrating points. The following theorem then provides a systematic way of 
doing so. 

Theorem 5. Assume the setup of Theorem 3. Let the nonzero components ofx have magnitude Xmag cind let 
h+ be as defined in (55). Further, let rgocp+ = liiUn^oo ''°°^^ and Xmag = lim„^oo -7^- ^l^o, let ^(/^"*) 
be as defined in (94). Let a = ^ and (3^ = - be fixed. Consider the following 

S{0[f-\ei^-^) = lim ^f^ = - f/3+./Te"^''^"'^^^-^"' 
n-s>oo n I V 27r 
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Ed,.,.^ l-Pti ^(^'■>^(2l^ - 1)) 



e i-/9u 






Q+ I \/2erftnv{2— 



gifeas) 



+ ^^ ^JI^) + ^2 - 1 + P^ 




nlaUeas)^ _ y Erdep+ _ (sc) _ (sc) q/aifeas) nifeas). (sc) /-, _ Afeas). 

^\"2 )~ ^^^ /— ~ ' socp+ ^mag'~'\"l ^"2 ) ' ^magX'- "2 ) 



^_gijeas) _^ _^ 



(-). ^.(-) lB^JJ-,~^''fi"'^'^i^-'^^ 



socp+ ' "^mag I Pw \/ o 



1 _ ffifeas) 

Fiei^^-'^) = V2erfinvi2 \ \^ - 1) 

I- Pw 

G(4^'^''')) = V2eryi«v(2^^-^| 1). (109) 

Pw 



Let the pair f^J^ , ^2 ) ^^ ^^^ solution of the following two equations 






G{9'^ ''''') + 



^2 + (l_0p-))(a,M^2 



mag J 

(feas)-. 



FieY"""'') 



(110) 



(sc) 

Then the feasibility breaking point for Xmag can be determined as the solution of 

lim :^i^ = X2 + (1 - e^\xtl?\{a - D{e^\e^^)) - R{e^^) 

n— >oo -i/n ' V 



.(^2 + (1 _ 4/-))(.M)2) ^(^2^^^ ) _ ^(^(/e-)) = 0. (Ill) 

3^ mag 



Proof Follows from Theorem 3 based on the discussion presented above and a combination of (94), (100), 
and (108). D 

Remark 1: It is relatively easy to see that 0)^^'^'^' = -^^ which somewhat simplifies the expression for 

D(^} "''^ ^2 )■ For the completeness though we chose to present the results in the above theorem in a 
general form. 

Remark 2: Another way to deal with the unboundedness (infeasibility) is to look at (68) and recognize 
that (a^e„_,_ — ||g||2 + ddep+) needs to be negative so that (57) is bounded. We presented the results in 
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Theorem 3 by assuming that all relevant parameters are such that (57) is bounded. Instead one could actually 
characterize them through one of the above approaches. However, we thought that it would complicate the 
presentation and opted for the current exposition. 

(sc) 

The results from the above theorem can be used to compute the breaking feasibility point for Xmag for 



particular values of /3+, a, a, and rsocp^ 
such results. 



In a part of the following subsection we will present a subset of 



3.2.2 Theoretical predictions 

In this subsection we present the theoretical predictions one can get based on the result of the previous 
sections. Similarly to what was done in Section 2.2.1 we will split the presentation of the results into several 
parts. 



1) 



E\\wdep+\\2 



"^''■^''p+"'^ as a function of x'mag 



To present this portion (as well as several others that will follow) of theoretical results we will as in 
Section 2.2.1 look at three regimes: 1) low a-, medium a-, and high a-regime. For each of the regimes 



we will show the theoretical results for ''''^"'" = 
a = 0.3 as a representative of the low a-regime, a 
and a 



E\\vJso. 



cp+112 



as a function of x 



{sc) 



mag- We wiU again take 
0.5 as a representative of the medium a-regime, 
0.7 as a representative of the high a-regime. We will consider rsocp+ = ^1°^ 



socp+ ~ " Y T+p^" 

For each of the a-regimes we will look at two different sub-regimes: low /3+- and high /3+ -regime (which 
based on results from [53] is equivalent to low p- and high p-regimes). For each of these two sub-regimes 
/3+ will be selected based on the curves obtained in [53] (or those obtained in [52]) in the following way. 



In the low /3+ sub-regime we will set p = 2 and r, 



{opt) 



socp-^ 



sub-regime we will set p 



.(opt) 
socp+ 



'^locp+ = ^V5 whereas in the high /3+ 
/"y^. At the same time from [53] we will 



have r 



socp+ 



^{opt) 
socp+ 



-- 3 and rsocp+ ■ 

J {a — aw)n where a^ and /3+ are such that (50) holds (we also recall on 



[53] where it was reasoned that the low j3 regime is selected so that the pair (a,/3+) is well below the 
fundamental characterization (50) whereas the high j3 regime is selected so that the pair (a, /3^) is closer to 
the fundamental characterization (50)). The values for ''"^^ = «^ ^"'^p+W'^ one can then get through 
the results of Theorem 4 for such (a,/3^) pairs and rj^cp+ ^•'^ shown in Figure 13 as functions of Xmag- 



As can be seen from Figure 13, the values of ''^''''^ 



E\\Waocp+\\2 



{sc) . 

converge to p as Xmag mcreases. 



II w, |L/o=E| 

dep+"2 



w |L/o as a function of x'^"'; signed 

SOCP+ 2 mag ' 



E||Wj IL/<i=E||w IL/o as a function of X™; signed X 

" Honi"? II socp+iig mag ^ 



eIIWh, 




/o=E|| w II ,/a as a function of x'"' ; signed x 

: socp+i 2 mag ' 



Feasibility breaking point p-3, x* '=1.7 



Feasibility breaking point p=2, x'^^' =0.65 



Figure 13: '^ = 

a = 0.5, right — a = 0.7 



^""^ — as a function of Xmag', rsocp+ 



i+p- 



^■, left — a = 0.3, center 
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This is of course in agreement with [53] where it was demonstrated that for rff^J^ one has p = "^''°^p+"'^ 
with overwhelming probability. Also, this is in a agreement with a similar conclusion made in Section 2.2.1. 

(sc) 

Furthermore, as it was the case in Section 2.2.1, the convergence is "faster" (or happens for smaller Xmag) 
for larger a. 

On the other hand one should observe from Figure 13 that for a = 0.7 the optimization problem in (57) 



(sc) 

is infeasible with overwhelming probability for Xmag below 
ss 0.65 in low /3+ regime. 



1.7 in high /3+ regime and for Xmag below 



2) ^hM± = <rr24-^^^^^^^9,rso.,+ ) ^^ ^f^^^^.^^ ^f^M 
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Similarly to what was discussed above (and is related to ||wsocp+||2 and ||wrfep+||2) one can determine 



^{dep) 



.(^c) 



the concentrating points of fobj+ and Cprim+ ^1^° ^^ functions of a^ma^- As in Section 2.2.1, to present these 
results we restrict ourselves to the medium a-regime, or in other words to a = 0.5. As in part 1) above we 
again choose rsocp+ = i^s^cp+ — ^a/t+^ ^^'^ consider low p = 2- and high p = 3- regime. The obtained 



i+P' 



results are shown in Figure 14. As can be seen from Figure 14 °p^ is larger for larger 
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Figure 14: ^^ = — Eni!i± as a function of 
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E\\vfdep+h 



E\\w 



socp-\- \\2 



as a function of Xmag; varying rsocp+ 



Another interesting set of results relates to possible variations in the rsocp+ that can be used in (4). The 
results that we presented above assume an optimal choice for rsocp+ (in a sense detined in [53]). Namely, 

)n where a^ and /3+ 



they assume that for a fixed pair (a, /3^) one chooses r 



socp+ 



S°Pi) 



socp+ 



a 



{a 



a-u 



are such that (50) holds. In the worst-case scenario (or in the generic scenario as we referred to it in [53]) 
one has that choice Tgf^J^ offers the minimal norm-2 of the error vector. However, such a scenario assumes 
particular x's which leaves a possibility that for a wide range of other x's the performance of the SOCP 
from (4) in the £2 norm of the error vector sense can be more favorable. Of course as shown in Figure (13) 
this indeed happens to be the case. On the other hand that also leaves an option that one can possibly choose 
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a different rsocp+ and get say a smaller norm-2 of the error vector for various different x. Below we present 
a few results in this direction. 

We will consider again only the medium or a = 0.5 regime. For two sets of different values of 



P> fsocp+ 



lopt) 
socp- 



, and /3+ we presented the results for 



g||Wdep+||2 



^W^'x'-p+W^ in Figure 13. 



In 



addition to that we now in Figure 15 show similar results one can get through Theorem 4 for two dif- 



ferent choices of r. 



ocpi 



To be more precise, for p = 2 we choose the same a and /3+ as in Figure 



and only vary rsocp+ over {a\/0.05an, a\/0.2ana\/0.6an}. Clearly, choice a\/0.05an is smaller than 



Jopt) 
' socp+ 

for p 



a\/0.2an whereas choice o"\/0.6an is larger than 



Aopt) 



socp~\ 



= a\/0.2an. On the other hand 
3 we choose the same a and /3+ as we have chosen ior p = 3 in Figure 13 and vary rsocp+ 
but this time over {a\/0.05an, cr\/0.1ano"\/0.5an}. Again, clearly, choice aV0-05an is smaller than 



Sopt) 
' socp+ 



aVO.lan whereas choice aVO.Gan is larger than 



„{opt) 
socp+ 



a\/0.2an. It is rather obvious but we 



mention for the completeness that the middle rsocp+ choices for both, p = 2 and p = 3, cases correspond 
to the center plot in Figure 13. Based on the results presented in Figure 15 one can then make the same two 
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Figure 15: ^"'^''J''+'^' = ^\\'^'^°j=p+\^2 as a function of x^ag for different rsocp+; left — p = 2, rsocp+ e 
{crVO.OSan, (Ty/^.2ana^jQ.Qan}; right — /O = 3, rsocp+ £ {cr\/0.05an, a^jQ.lana^/Q.han} 



observations that we made earlier related to the results presented in Figure 5. The first one is that Figure 15 
suggests that if rsocp+ is smaller than rJ^J^ then Il^s°gp+ll2 could be larger than the one that can be ob- 
tained for rg°^p^_. This actually happens to be the case. As in Section 2.2.1 we skip the details of this simple 



(opt) 



exercise. The second observation is that if rsocp+ is larger than rl^'j^ then for certain x 



E\\vfao. 



ep+l|2 



could 



„(op*) 



be smaller than the one that can be obtained for r\^cp+- ^W^ of course suggests that a choice of rsocp+ larger 
than ri^(.p+ could be more favorable in certain applications and for a particular measure of performance. 

However, if one has no a priori available knowledge about x then adapting rsocp+ beyond rs^cp+ would be 

hard. 

4) ^ = <^^V(-'g^"-"^''^--''+) as a function ofxt%; varying Tsocp+ 

Similarly to what was done above in part 2) one can also detemiine the theoretical predictions for 

Efobj+ 



(dep) 



medium a 



'snsi± for a varying rsocp+- As in parts 2) and 3) above, we restrict our attention only to the 
0.5 regime. We also assume exactly the same scenarios as in part 3). The obtained results are 
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shown in Figure 6. As in part 2) Figure 16 shows that '^^ is larger for larger p. On the other hand it also 
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Figure 16: ^^ = ^CT2+('''S,h^"^a„rsoc,+) ^^ ^ function of x^Jg for different rsocp+; left — p = 2, 
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shows that p-^ decreases as Vsocp^ increases. This also follows rather trivially from the structure of (47) 
or in a way discussed in the corresponding part of Section 2.2.1. 

As in Section 2.2. 1 we conducted massive numerical experiments and again found that the results one 
can get through them are in a firm agreement with what the presented theory predicts. In the next subsection 
we present a sample of the results obtained through the conducted numerical experiments. 



3.2.3 Numerical experiments 

As in earlier subsection we will split the presentation of the numerical results in several parts. The numerical 
results that we will present below are obtained by running the SOCP from (47). To demonstrate the precision 
of our technique we will in parallel show the results obtained by running (57). To make scaling simpler in 
all our numerical experiments we again set a = 1. 



^^ g||wrfep+l|2 ^^^ ^ll^^°^p+lb as functions of x 



(sc) 
mag 



In this part we will show the numerical results that correspond to the theoretical ones given in part 1) in 
the previous subsection. We will restrict our attention again only on the medium or a = 0.5 regime (in a 
later section we will show the results one can get for a = 0.7 regime). We then set all other parameters as in 
the center plot of Figure 13 (these parameters are of course different depending if we are considering p = 2 
or p = 3; below we will consider both of them). 

a) Low (a, /3^) regime, p = 2 

We first consider the p = 2 scenario. As mentioned above in our experiments we set a = 0.5, rsocp+ = 

-r^a. We then 



\/0.2an, and (as shown in [53]) /3^ such that (a+, /3+) satisfy (50) and a^ 



(sc) 

ran (47) 300 times with n = 600 for various Xmag- In parallel we ran (16) for the exact same parameters 

with only one difference; namely we ran (57) with n = 2000. The obtained results for H^'iogp+lla ^j^j 

''^''''^ are shown on the left-hand and right-hand side of Figure 23, respectively (given our assumption 

that cr = 1 "'•^'i^p+»^ and ^^'^"""''"''"^ are of course just -©Hvifrfep+lb and £'||wsocp+||2> respectively). We 
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also show in Figure 23 the corresponding theoretical predictions obtained in the previous subsection. 
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Figure 17: Experimental results for ^"^^7^+"" and ^"'^'';^+"' as a function of x^^%; p 
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b) High (a, /3+) regime, p = 3 

We also conducted a set of experiments in the so-called "high" (a,/3+) regime. We used exactly the 
same parameters as in low (a, /?+) except that we changed p from 2 to 3. Consequently we chose rsocp+ = 
VO.lan and /3+ such that (a+, /3+) satisfy (50) and a+ = jf^a. As above we ran 300 times each (47) 
and (57). We ran (47) with n = 600 and (57) with n = 2000. The numerical results obtained for p = 3 
together with the theoretical predictions are again shown in Figure 23. From Figure 23 we observe a solid 
agreement between the theoretical predictions and the results obtained through numerical experiments. 

2) :%± and K^"rl+i-'^^^^-^a„r^..,+) ^^^^^^^^^^ ^_^^M 



^/n 



y/n 



In this part we will show the numerical results that correspond to the theoretical ones given in part 2) in 
the previous subsection. We then set all parameters as in Figure 13 (these parameters are exactly the same as 
in experiments whose results we just presented above). Of course we again distinguish two cases: p = 2 and 
p = 3. For both p = 2 and p = 3 we ran 300 times each, (47) and (57) and again we ran (47) with n = 600 

and (57) with n = 2000. The numerical results that we obtained for °^^ and 



are shown in Figure 18. We again observe a solid agreement between the theoretical predictions and the 
results obtained through numerical experiments. 



3) 



Sljw 



dep+H2 



and 



■E||w3oep+||2 



as functions o/xmag,' varying rsocp+ 



In this part we will show the numerical results that correspond to the theoretical ones given in part 3) in 
the previous subsection. These results relate to possible variations in the rsocp+ that can be used in (47). We 
then set all other parameters as in Figure 15 (these parameters are of course again different depending if we 
are considering p = 2 or p = 3). 

a) Low (a, /3^) regime, p = 2 

We first consider the p = 2 scenario. As in part 1) of this subsection we set a = 0.5 and choose /3^ as 
in part 1). However, differently from part 1) we now consider two different possibilities for rsocp+, namely 
VO.OSan and rsocp+ 



^socp+ 

In parallel we ran (57) with n 



\/0.6an. We then ran (47) 300 times with n 



600 for various Xmag- 



2000. The obtained results for 



_E||w 



socp-\- II 2 



and '''"'i^p+»^ are shown on 
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Figure 18: Experimental results for ^^ and ^CT2+i-'S,h^^a„rsocp+) ^^ ^ function of xtJg', P = 2, 
rsocp+ = V0.2an; p = 3, rsocp+ = VO.lan; left — SOCP from (47); right — (57) 



the left-hand and right-hand side of Figure 19, respectively. We also show in Figure 19 the corresponding 
theoretical predictions obtained in the previous subsection. 



E||w IL/o as a function of x''^'^' 
socp+"2 mag 



2.5 

2 

e 

Q. 

8 1.5 

L 

LU 

1 

0.5 



sometimes infeasible 




■■■ r =(0.05m)"^, theoretical 

socp+ /,„ 

r =(0.6m)^' , theoretical 



socp- 



,^1'2 



r =(0.05m) , experimental 
r =(0.6m), experimental 

5QCp+ '^ 



10 12 14 16 



x'"-'=x In" 



Ell w^ IL/a as a function of x'^°' 

" dep+"2 mag 



2.5 



-g 1.5 



sometimes unbounded 

/ 


^,,,->.-'-L.I. 




^^S^,^'^ 


/ 


-'■ r =(0.05m)^'^ theoretical 

SOCp+ . ,o 

— r ={0.05m) , experimental 

SOCp+ J (o 

I.I. '■socp+^^^-^'^^ ' e>*pen mental 



6 8 10 12 14 16 

mag mag 



Figure 19: Experimental results for ^\\^^°-vA\2 ^^^ EW^^^^^^W^ ^^ ^ function of x^^ P = 2; rsocv+ e 
{VO.OSan, V0.6an}; left — SOCP from (47), right — (57) 



h) High {a, j3^) regime, p = 3 

We also consider the p = 3 scenario. As above, we set a = 0.5 and choose /3+ as in part 1) of this 
subsection. Everything else remain the same as in p = 2 case except the way we vary rsocp+- This time we 
consider (as in part 3) of the previous section when p = 3 case was considered) rsocp+ = \/0.05an and 
rsocp+ = \/0-5an. As usual (47) was run 300 times with n = 600 for various x^ag. In parallel we ran (57) 



300 times with n = 2000. The obtained numerical results for ^^^ and ^^prL+i''^sM,x^ai„rsocp+) 



y/n 



Vn 



as well 
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Table 1: Experimental results for the noisy recovery through SOCP; rsocp+ 
run 300 times with n = 600; (57) was run 300 times with n = 2000 



VO.OSm, CT = 1; (47) was 







(sc) 
^mag 


0.1 


0.3 


0.5 


0.7 


1 


2 


p = 


2 


'^socp+ 


= V0.05n 


# of successes (47) 


269 


274 


286 


294 


296 


300 


p = 


2 


'^socp+ 


= V0.05n 


# of successes (57) 


268 


278 


281 


287 


299 


299 


p = 


3 


^socp+ 


= VO.OSn 


# of successes (47) 


277 


287 


292 


296 


299 


300 


p = 


3 


^socp+ 


= V0.05n 


# of successes (57) 


270 


274 


288 


294 


299 


299 



as the corresponding theoretical predictions obtained in the previous subsection are shown on the left-hand 
and right-hand side of Figure 20, respectively. We again observe a solid agreement between the theoretical 
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{V0.05an, V0.5an}; left — SOCP from (47), right — (57) 

predictions and the results obtained through numerical experiments. As in Section 2.2.2 small glitches that 
happen in large Xmag regime could have been fixed by choosing a larger n. We again purposely chose a 
smaller n to show that results are fairly good even when n is not very large. In fact, even a smaller n than 
the one that we have chosen would work quite fine. 

Another observation related to Figures 19 and 20 (and several figures that will follow) is in place. For 



socp+ 



VO.OSan and roughly speaking Xmdg < 2 it may happen that (47) is on occasion infeasible and 



„(*c) 



that (57) is unbounded. From a theoretical point of view this should not happen for any Xmdg- However, as 
discussed in Section 3.2.1, a = 0.5 is in a sense a border line choice for universal feasibility. On the other 
hand, since all these claims are of "with overwhelming probability" type it may sometimes happen that even 
when a = 0.5 (47) is infeasible and (57) is unbounded. In Table 1 we show the number of our experiments 
for which everything worked fined, i.e. for which (47) turned out to be feasible and (57) turned out to be 
bounded (we restrict only to what we call interesting region, which for this example we found to be roughly 



(sc) 
fJmag 



< 2). We refer to such a number as the number of successes. The results in rsocp+ = V0.05an regime 
shown in Figures 19 and 20 are averaged over the feasible instances of (47) and the bounded instances of 

(57). 
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4) —^ and — p""'+ <^'S,^mag,rsocp+ ^g j-j^fj^^^fjg ofxmdg; varying rsocp+ 

In this part we will show the numerical results that correspond to the theoretical ones given in part 4) in 

the previous subsection. These results relate to behavior of i^"*" and — Eninl — ' ' ""''" ""''^ — when one 
varies rsocp+ in (47). We again consider p = 2 or p = 3. The observations made above that relate to the 
occasional feasibilities do apply to the results presented in this part as well. 

a) Low {a, (3^) regime, p = 2 

The setup that we consider is exactly the same as the one considered in part 3a) of this subsection. 
We set a = 0.5, chose /3+ as in part 1), and considered two different possibilities for rsocp+, namely 

rsocp+ = V0^05^ and rsocp+ = Vol^. The obtained results for ^^ and ^V.^+^'^'g'y-«'^-°-^+^ 



\/n 



y/n 



are shown on the left-hand and right-hand side of Figure 21, respectively. The corresponding theoretical 
predictions obtained in the previous subsection are also shown in Figure 21. 
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Figure 21: Experimental results for p~^ and — eih^I — ' ' """'" """'''^ as a function of Xmag', p 
rsocp+ G {\/0.05an, VO.Gan}; left — SOCP from (47), right — (57) 

b) High {a, /3^) regime, p = 3 

The setup that we consider is exactly the same as the one considered in part 3b) of this subsection. 
We set a = 0.5, chose /3+ as in part 1), and considered two different possibilities for rsocp+, namely 
rsocp+ = V0.05an and rsocp+ = VO.San. The obtained results for ^\\'^^°jp+\\'^ and ^"'^'';''+"' are shown 
on the left-hand and right-hand side of Figure 22, respectively. The corresponding theoretical predictions 
obtained in the previous subsection are also shown in Figure 22. We again observe a solid agreement 
between the theoretical predictions and the results obtained through numerical experiments. 



3.2.4 Numerical experiments — feasibility 

In this section we will present a couple of numerical results that relate to the feasibility of (47) or unbound- 
edness of (57). 
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In this part we will show the numerical results that correspond to the theoretical ones given in part 1) in 
the previous subsection. We will restrict our attention to a = 0.7 regime (we recall that earUer in Section 
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Figure 22: Experimental results for °^'^ and — Eii£i± — ' ' ""''" """'''^ as a function of Xmag', p = 3; 
rsocp+ G {V0.05an, \/a5cm}; left — SOCP from (47), right — (57) 



3.2.3 we showed the corresponding results one can get for a = 0.5 regime). We then set all other parameters 
as in the plot on the right hand side of Figure 13 (these parameters are of course different depending if we 
are considering p = 2 or p = 3; below we will consider both of them). 

a) Low (a, (3^) regime, p = 2 

We first consider the p = 2 scenario. We set a = 0.7, rsocp-\ 
in [53]) /3+ such that («+,/?+) satisfy (50) and a^ 



i+P^ 



\/0.2an, and (as shown 



i+p- 



■a. We then ran (47) 100 times with n = 800 



for various Xmag- In parallel we ran (57) for the exact same parameters with only two differences; namely 

h and ^1'^"-^+"^ are shown on 



£^||w, 



socp-\- 



we ran (57) 200 times with n = 4000. The obtained results for 

the left-hand and right-hand side of Figure 23. We also show in Figure 23 the corresponding theoretical 

predictions obtained earlier. 

b) High {a, 13:^) regime, p = 3 

We also conducted a set of experiments in the so-called "high" (a,/3^) regime. We used exactly the 
same parameters as in low {a, /3+) except that we changed p from 2 to 3. Consequently we chose rsocp+ = 
VO.lon and /3+ such that (a+,/3+) satisfy (50) and a+ = yt"^"- As above we ran 100 times (47) and 
200 times (57). Also as above, we ran (47) with n = 800 and (57) with n = 4000. The numerical results 
obtained for p = 3 together with the theoretical predictions are again shown in Figure 23. From Figure 23 
we observe a solid agreement between the theoretical predictions and the results obtained through numerical 
experiments. 

(sc) 

Remark: We do mention that in a range of Xmag close to the theoretical "breaking feasibility point" not 
all instances of (47) were feasible and not all instances of (57) were bounded (instead an overwhelming 
majority of them was). The results that are shown in Figure 23 are obtained by averaging over the feasible 
instances of (47) and the bounded instances of (57). 

2) Feasibility/boundedness probability 

In this part we show how the number of feasible instances of (47) and the number of bounded instances 

(sc) 

of (57) change as Xmag changes. We set a = 0.7 and considered the p = 3 regime or in other words 
"high" (q,/3+) regime. As above we chose rsocp+ = VO.lan and /3+ such that (a^,/?^) satisfy (50) 
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Figure 23: Experimental results for ^"^''"'^p+H^ and ^"'^'';^+"' as a function of x^ag; a = 0.7; p = 2, 
rsocp+ = V0.2an; p = 3, rsocp+ = \/Olcm; left — SOCP from (47), right — (57) 



and a^ 
with n 



j^a. We then ran each of (47) and (57) 200 times. We ran (47) with n 



1000 and (57) 



10000. For a range of Xmag we recorded the fraction of instances where (47) was feasible. We 

(sc) 

refer to such a fraction as Psocp+- Simultaneously, for the same range of Xmag we recorded the fraction 
of instances where (57) was bounded. We refer to such a fraction as Pprim+- The numerical results along 
with the theoretical prediction for the feasibility breaking point are shown in Figure 24. From Figure 24 we 
observe a solid agreement between the theoretical predictions and the results obtained through numerical 
experiments. 



4 Discussion 

In this paper we considered "noisy" under-determined systems of linear equations with sparse solutions. 
We looked from a theoretical point of view at polynomial-time second-order cone programming (SOCP) 
algorithms. More precisely, we looked at a general framework developed for characterization of such al- 
gorithms in [53]. Within the framework we then considered what we referred to as the SOCP's problem 
dependent performance. We established the precise values of the norm- 2 of the error vector for a wide class 
of unknown sparse vectors. We also provided a characterization of several important parameters that appear 
in solving of an SOCP. 

Many further developments are possible (one can make essentially the same conclusion for the general 
framework developed in [53] as well as for the analysis of the LASSO algorithms presented in [52]). Any 
problem dependent scenario that can be solved in the so-called noiseless case through the mechanisms devel- 
oped in [57] and [55] can now be handled in the noisy case as well. For example, quantifying performance 
of SOCP or LASSO optimization problems in solving "noisy" systems with special structure of the solution 
vector (block-sparse, box- and binary-sparse, partially known locations of nonzero components, low-rank 
matrix, just to name a few), "noisy" systems with noisy (or approximately sparse)) solution vectors can then 
easily be handled to an ultimate precision. In several forthcoming papers we will present some of these 
applications. 
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